Create an Undistorted Top-Down View of Camera Image

I have a fixed camera mounted on a wall viewing a rectangular lawn at an angle. My goal is to obtain an undistorted, top-down view of the lawn.

I have an image from the camera as a python numpy array which looks like this:

raw camera image

I use an inverse matrix with skimage.transform.warp to correct the image to a top down view:

top down distorted

This works perfectly, however the camera lens introduces barrel distortion.

Seperately, I can correct the distortion with a generated lookup table using skimage.transform.warp_coords and passing a simple undistort callable function based on the algorithm described here. The image is then generated using scipy.ndimage.map_coordinates:

undistorted camera view

These 2 processes work individually, but how do I combine them to create an undistorted top-down view, without creating an intermediate image?

I could run each point in the lookup table through the matrix to create a new table, but the table is massive and memory is tight (Raspberry Pi Zero).

I would like to define the undistortion as a matrix and just combine the 2 matrices, but as I understand it, the projective homography matrix is linear but undistortion is non-linear, so this can't be done. I can't use OpenCV due to resource constraints, and the calibration procedure involving multiple chessboard images is impractical. Currently, I calibrate by taking 4 lawn corner points and generate the matrix from them, which works well.

I would have anticipated that this is a common problem in Computer Vision but can't find any suitable solutions.

from Create an Undistorted Top-Down View of Camera Image