OpenCV unsatisfying results when finding Homography from ORB feature detection

Even though the ORB Feature Matching seems quite solid and i only take the 20 best matches for cv.findHomography, the resulting polyline is terrible. Note that in the results shown in the attached image, the top right image is a video stream. Therefor the variation in results matched. Is there a library that could be used to receive better results? Or am I doing any major mistakes in my code?

    # des1 & des2 are created with cv.ORB_create(10000, 1.2, nlevels=8, edgeThreshold=5)

    kp2, des2 = orb.detectAndCompute(gray, None)
    matches = bf.knnMatch(des1, des2, k=2)

    good = []
    for m, n in matches:
        if m.distance < 0.75 * n.distance:
            good.append(m)

    matches = sorted(good, key=lambda x: x.distance)
    src_pts = np.float32([kp1[m.queryIdx].pt for m in matches[:20]]).reshape(-1, 1, 2)
    dst_pts = np.float32([kp2[m.trainIdx].pt for m in matches[:20]]).reshape(-1, 1, 2)
    M, mask = cv.findHomography(dst_pts, src_pts, cv.RANSAC, 5.0)
    matchesMask = mask.ravel().tolist()

    h = src_pts.max(0)[0][1] - src_pts.min(0)[0][1]
    w = src_pts.max(0)[0][0] - src_pts.min(0)[0][0]
    pts = np.float32([[0, 0], [0, h - 1], [w - 1, h - 1], [w - 1, 0]]).reshape(-1, 1, 2)

    dst = cv.perspectiveTransform(pts, M)

    img3 = None
    img3 = cv.drawMatchesKnn(img1, kp1, gray, kp2, good, img3, flags=cv.DrawMatchesFlags_NOT_DRAW_SINGLE_POINTS)
    img3 = cv.polylines(img3, [np.int32(dst)], True, (0, 0, 255), 3, cv.LINE_AA)

    # Code for showing img3 would follow
    

from OpenCV unsatisfying results when finding Homography from ORB feature detection