MKMapCamera doesn't zoom to correct altitude

When I'm setting an MKMapCamera with a specific altitude on my map view (MapKit) it doesn't zoom to the correct altitude sometimes. I think it has something to do with the map not being fully loaded so it stops higher (950m or so) instead of the altitude I set (about 280m).

Initially I noticed the issue when I first loaded the map but it seems more related to the lower altitudes. Higher altitudes seems to work OK.

Here's a video demonstrating the problem: https://streamable.com/644l1 In the video I'm setting the same camera twice.

The code for setting the camera:

let distance = currentHole!.teeCoordinate.distance(to: currentHole!.greenCoordinate)
let altitude = Double(distance) * 2.5

let camera = MKMapCamera(
    lookingAtCenter: currentHole!.centerCoordinate(),
    fromDistance: altitude,
    pitch: 0.0,
    heading: currentHole!.teeCoordinate.bearing(to: currentHole!.greenCoordinate) - 20
)
mapView.setCamera(camera, animated: true)

I also tried to use something like:

UIView.animate(withDuration: 1.0, animations: { () -> Void in
    self.mapView.camera = camera
}, completion: { (done) -> Void in
    print("Animation complete")
})

to do the animation instead. It works better (not perfect) when setting the duration to something very high, like 10 seconds or so.

Any ideas on what might be the issue here?

UPDATE:

It seems to only happen with "Satellite Flyover" maps. Satellite is fine.

from MKMapCamera doesn't zoom to correct altitude