Tuesday, 23 January 2024

reconstruction of image shows either black either square borders

I have trained two models (forward and backward).

(The input to the models are images of type uint8, so I am dividing by 255)

After predicting on each model, I receive two arrays:

forward = np.load('f.npy')
backward = np.load('b.npy')

I also must use an image tiles_M in order to follow these equations:

p1 = ( 1.0 / abs(forward - tiles_M/255.) ) / ( (1.0 / abs(forward - tiles_M/255.)) + (1.0 / abs(backward - tiles_M/255.)) )
p3 = ( 1.0 / abs(backward - tiles_M/255.) ) / ( (1.0 / abs(forward - tiles_M/255.)) + (1.0 / abs(backward - tiles_M/255.)) )

Note, that, I divide tiles_M by 255 (the same I did in inputs for training the models) since it is an uint8 image.

Then, the prediction must use this equation:

pred = p1 * forward + p3 * backward

The problem, is when I try to reconstruct the image, I receive a black image (all zero values).

If I normalize pred : pred = normalize_arr(pred) I receive this image here

I have tried various ways to normalize either pred or p1, p2, forward, backward but now works as expected.

Now the interesting part comes from this.

If I use this equation (which is wrong and I accidentally typed at some point!):

p1 = ( 1.0 / abs(forward ) ) / ( (1.0 / abs(forward - tiles_M)) + (1.0 / abs(backward - tiles_M)) )
p3 = ( 1.0 / abs(backward) ) / ( (1.0 / abs(forward - tiles_M)) + (1.0 / abs(backward - tiles_M)) )

so, no tiles_M scaling and no subtraction from tiles_M in the numerator, I receive this correct image!!!

The equation is:

this

here

You can find the data here.

This is the code:

import numpy as np
import cv2
from PIL import Image

def normalize_arr(arr):
  the_min = arr.min()
  the_max = arr.max()
  the_max -= the_min
  arr = ((arr - the_min)/the_max) * 255.
  return arr.astype(np.uint8)

def extract_tiles(size, im):
    im = im[:, :, :3]
    w = h = size
    idxs = [(i, (i + h), j, (j + w)) for i in range(0, im.shape[0], h) for j in range(0, im.shape[1], w)]
    tiles_asarrays = []
    count = 0
    for k, (i_start, i_end, j_start, j_end) in enumerate(idxs):
        tile = im[i_start:i_end, j_start:j_end, ...]
        if tile.shape[:2] != (h, w):
            tile_ = tile
            tile_size = (h, w) if tile.ndim == 2 else (h, w, tile.shape[2])
            tile = np.zeros(tile_size, dtype=tile.dtype)
            tile[:tile_.shape[0], :tile_.shape[1], ...] = tile_
        
        count += 1
        tiles_asarrays.append(tile)
    return np.array(idxs), np.array(tiles_asarrays)


IMG_WIDTH = 32

# Load arrays
forward = np.load('f.npy')
backward = np.load('b.npy')
tiles_M = np.load('tiles_M.npy')

# Weighting params
p1 = ( 1.0 / abs(forward - tiles_M/255.) ) / ( (1.0 / abs(forward - tiles_M/255.)) + (1.0 / abs(backward - tiles_M/255.)) )
p3 = ( 1.0 / abs(backward - tiles_M/255.) ) / ( (1.0 / abs(forward - tiles_M/255.)) + (1.0 / abs(backward - tiles_M/255.)) )

# works but wrong equation and no tiles_M scaling
# p1 = ( 1.0 / abs(forward ) ) / ( (1.0 / abs(forward - tiles_M)) + (1.0 / abs(backward - tiles_M)) )
# p3 = ( 1.0 / abs(backward) ) / ( (1.0 / abs(forward - tiles_M)) + (1.0 / abs(backward - tiles_M)) )


pred = p1 * forward + p3 * backward
#pred = normalize_arr(pred)

# Load original image
img = cv2.imread('E2.tif',
                 cv2.IMREAD_UNCHANGED)
img = cv2.resize(img, (1408, 1408), interpolation=cv2.INTER_AREA)
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

# create tiles 
idxs, tiles = extract_tiles(IMG_WIDTH, img)

# Initialize reconstructed array
reconstructed = np.zeros((img.shape[0],
                          img.shape[1], 
                          img.shape[2]),
                          dtype=np.uint8)

# reconstruct
for tile, (y_start, y_end, x_start, x_end) in zip(pred, idxs):
    y_end = min(y_end, img.shape[0])
    x_end = min(x_end, img.shape[1])
    reconstructed[y_start:y_end, x_start:x_end] = tile[:(y_end - y_start), :(x_end - x_start)]
    
# create image from array
im = Image.fromarray(reconstructed)
im = im.resize((1429, 1416))
im.show()


from reconstruction of image shows either black either square borders

No comments:

Post a Comment