MediaCodec.VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING constantly jumping

I have this problem which honestly I don't know how to solve, I'm using a SurfaceView to render content. The video is a letterbox type video where margins needs to be chopped of from the video in order to make it fill the screen and be in correct aspect ratio.

I have tried 3 different things till now:

1. Put the SurfaceView in the center of an over-sized FrameView (this worked but on some devices it caused video artefacts)

2. Use a TextureView and a matrix to scale, this works but on some devices I'm getting a green screen or very poor performance, beside Google suggest that using SurfaceView is a better approach.

3. My preferred (and probably most correct) approach is to use a SurfaceView and set media codec to: VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING something like this:

                    @Override
                    public void onOutputFormatChanged(@NonNull MediaCodec mediaCodec, @NonNull MediaFormat mediaFormat) {
                        android.util.Log.d(TAG, "Output format changed!");
                        mediaCodec.setVideoScalingMode(MediaCodec.VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING);
                    }

This does work, as expected however on some devices (which I can't reproduce on any of my tests), this happens:

https://www.youtube.com/watch?v=FFOafPVgADQ

https://www.youtube.com/watch?v=38wW_AeNnTg

The MediaCodec is created like:

MediaCodec.createDecoderByType("video/avc");

I have tried using, which seems to fix the problem, but has very poor performance (as it's done by the CPU rather then GPU)

MediaCodec.createByCodecName("OMX.google.h264.decoder");

Any idea why is that jumping happening? Am I right to think that it's a broken video driver and probably I won't be able to fix it?

from MediaCodec.VIDEO_SCALING_MODE_SCALE_TO_FIT_WITH_CROPPING constantly jumping