Hemant Vishwakarma: Rectangle detection inaccuracy using approxPolyDP() in openCV

As part of a program , which contains a series of images to be processed, I first need to detect a green coloured rectangle first. I'm trying to write a program that doesn't use colour masking, since the lighting/glare on the images will make it difficult to find the appropriate HSV ranges.

(p.s: I already have two questions based on this "program", but this one is unrelated to those. It's not a follow up, I want to address a separate issue.)

I used the standard rectangle detection technique: making use of findContours() and approxPolyDp() methods. I added some constraints that got rid of unnecessary rectangles (like aspectRatio>2.5: since my desired rectangle is clearly the "widest" and area>1500, to discard random small rectangles) .

import numpy as np
import cv2 as cv

img = cv.imread("t19.jpeg")

width=0 
height=0

start_x=0 
start_y=0
end_x=0 
end_y=0


output = img.copy()
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)


#threshold
th = cv.adaptiveThreshold(gray,255,cv.ADAPTIVE_THRESH_GAUSSIAN_C,cv.THRESH_BINARY,9,2)

cv.imshow("th",th)



#rectangle detection

contours, _ = cv.findContours(th, cv.RETR_TREE, cv.CHAIN_APPROX_NONE)

for contour in contours:

    approx = cv.approxPolyDP(contour, 0.01* cv.arcLength(contour, True), True)
    
    cv.drawContours(img, [approx], 0, (0, 0, 0), 5)
    
    x = approx.ravel()[0]
    y = approx.ravel()[1]

    x1 ,y1, w, h = cv.boundingRect(approx)
    a=w*h    
    if len(approx) == 4 and x>15  :
            
        aspectRatio = float(w)/h
        if  aspectRatio >= 2.5 and a>1500:          
          print(x1,y1,w,h)
          width=w
          height=h   
          start_x=x1
          start_y=y1
          end_x=start_x+width
          end_y=start_y+height      
          cv.rectangle(output, (start_x,start_y), (end_x,end_y), (0,0,255),3)
          cv.putText(output, "rectangle "+str(x1)+" , " +str(y1-5), (x1, y1-5), cv.FONT_HERSHEY_COMPLEX, 0.5, (0, 0, 0))
          
cv.imshow("op",output)

print("start",start_x,start_y)
print("end", end_x,end_y)
print("width",width)
print("height",height)

It is working flawlessly for all the images, except one:

I used adaptive thresholding to create the threshold, which was used by the findContours() method. I tried displaying the threshold and the output , and it looks like this:

The thresholds for the other images also looked similar...so I can't pinpoint what exactly has gone wrong in the rectangle detection procedure.

Some tweaks I have tried:

Changing the last two parameters in the adaptive parameters method. I tried 11,1 , 9,1, and for both of them, the rectangle in the threshold looked more prominent : but in this case the output detected no rectangles at all.
I have already disregarded otsu thresholding, as it is not working for about 4 of my test images.

What exactly can I tweak in the rectangle detection procedure for it to detect this rectangle?

I also request , if possible, only slight modifications to this method,, and not some entirely new method. As I have mentioned, this method is working perfectly for all of my other test images, and if the new suggested method works for this image and fails for the others, then Ill find myself back here asking why it failed.

Edit: The method that abss suggested worked for this image, however failed for:

Other test images:

from Rectangle detection inaccuracy using approxPolyDP() in openCV