How to fix pinch zoom focal point in a custom view?

For my question I have prepared a very simple test app at Github.

For simplicity I have removed flinging, scroll constraints and edge effects (which actually work well in my real app):

So the custom view in my test app only supports scrolling:

mGestureDetector = new GestureDetector(context,
        new GestureDetector.SimpleOnGestureListener() {
    @Override
    public boolean onScroll(MotionEvent e1, MotionEvent e2, float dX, float dY) {
        mBoardScrollX -= dX;
        mBoardScrollY -= dY;

        ViewCompat.postInvalidateOnAnimation(MyView.this);
        return true;
    }
});

and pinch zooming with 2 fingers (the focus is broken though!):

mScaleDetector = new ScaleGestureDetector(context,
        new ScaleGestureDetector.SimpleOnScaleGestureListener() {
    @Override
    public boolean onScale(ScaleGestureDetector scaleDetector) {
        float focusX = scaleDetector.getFocusX();
        float focusY = scaleDetector.getFocusY();
        float factor = scaleDetector.getScaleFactor();

        mBoardScrollX = mBoardScrollX + focusX * (1 - factor) * mBoardScale;
        mBoardScrollY = mBoardScrollY + focusY * (1 - factor) * mBoardScale;
        mBoardScale *= factor;

        ViewCompat.postInvalidateOnAnimation(MyView.this);
        return true;
    }
});

Finally, here the code drawing the scalled and offsetted game board Drawable:

@Override
protected void onDraw(Canvas canvas) {
    super.onDraw(canvas);

    canvas.save();
    canvas.scale(mBoardScale, mBoardScale);
    canvas.translate(mBoardScrollX / mBoardScale, mBoardScrollY / mBoardScale);
    mBoard.draw(canvas);
    canvas.restore();
}

If you try running my app, you will notice that while scaling the game board with two finger pinch zoom gesture works, the zoom focal point jumps around.

My understanding is that while scaling the Drawable I need to pan it by adjusting the mBoardScrollX and mBoardScrollY values - so that the focus point stays at the same point in the game board coordinates. So my calculation is -

The old position of that point is:

-mBoardScrollX + focusX * mBoardScale
-mBoardScrollY + focusY * mBoardScale

and the new position would be at:

-mBoardScrollX + focusX * mBoardScale * factor
-mBoardScrollY + focusY * mBoardScale * factor

By solving these 2 linear equations I get:

mBoardScrollX = mBoardScrollX + focusX * (1 - factor) * mBoardScale;
mBoardScrollY = mBoardScrollY + focusY * (1 - factor) * mBoardScale;

However that does not work!

To eliminate any errors I have even tried hardcoding the focal point to the middle of my custom view - and still the game board center sways around while scaling:

float focusX = getWidth() / 2f;
float focusY = getHeight() / 2f;

I think I am missing something minor, please help me.

I would prefer to find a solution without using Matrix, because I believe that something really minor is missing in the above calculations. And yes, I have already studied a lot of comparable code, including the PhotoView by Chris Banes and the InteractiveChart example by Google.

from How to fix pinch zoom focal point in a custom view?