Fill holes/blocks in masked frames

I want to fill the "holes" in my masked image based on certain criteria. I have the following raw frame:

Next, I apply a mask over this frame which gives masked:

Now I want to divide the frame into 16x16 blocks. In order to define block(x,y), we define a set of pixels in the x-th and y-th block in vertical and horizontal direction. A block is defined as non-field (field) if the percentage of black pixels within the block is larger (smaller) than 0.5. The code looks as follows:

def blocked_img(img, x_pixels, y_pixels):
   blocked = img.copy()
   gray = cv2.cvtColor(blocked, cv2.COLOR_BGR2GRAY)

   for x in range(0, blocked.shape[1], x_pixels):
      for y in range(0, blocked.shape[0], y_pixels):
         block = gray[y:y+y_pixels, x:x+x_pixels]
         if (cv2.countNonZero(block) / block.size) < 0.5:
            # non-field
            cv2.rectangle(blocked, (x, y), (x+x_pixels, y+y_pixels), (0,255,255), 2)
         else:
            # field
            break

    return blocked

masked.shape returns the following:

 (720, 1280, 3)

So, I call the function blocked_img by using:

blocked = blocked_img(masked, 80, 45)     #  divide by 16

The output, blocked, looks as follows:

At the moment, my code is written that as soon as the 0.5-threshold is not reached, it loops to the next column. It can be seen that in the columns on the right, this method does not produce the desirable outcome as it stops prematurely (columns 9, 11, 12, 13, 16 to be precise)

I would like to continue the loop within the column if one the two following is in place:

• current block is non-field if previous block is non-field AND [next block is non-field OR next_next block is non-field]
• current block is non-field if [previous_previous block is non-field OR previous block is non-field] AND next block is non-field

Rewritten,

   block(x,y) = 1 if [block(x-1,y) = 1 and {block(x+1,y) = 1 or block(x+2,y) = 1}] or [{block(x-2, y) = 1 or block(x-1, y) = 1} and block(x+1, y) = 1]       # where 1 = non-field

Any idea how I can include this in my code? Thanks in advance!

from Fill holes/blocks in masked frames