Tuesday, 18 May 2021

Image augmentation in Keras mixed functional model

I've created a mixed model in Keras, creating weights for metadata and image data and then combining them for the classification. Here's the model:

Model: "model_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_5 (InputLayer)            [(None, 80, 120, 3)] 0                                            
__________________________________________________________________________________________________
xception (Functional)           (None, 3, 4, 2048)   20861480    input_5[0][0]                    
__________________________________________________________________________________________________
input_4 (InputLayer)            [(None, 10)]         0                                            
__________________________________________________________________________________________________
conv2d_9 (Conv2D)               (None, 3, 4, 8)      409608      xception[0][0]                   
__________________________________________________________________________________________________
dense_3 (Dense)                 (None, 4)            44          input_4[0][0]                    
__________________________________________________________________________________________________
global_average_pooling2d_1 (Glo (None, 8)            0           conv2d_9[0][0]                   
__________________________________________________________________________________________________
concatenate_1 (Concatenate)     (None, 12)           0           dense_3[0][0]                    
                                                                 global_average_pooling2d_1[0][0] 
__________________________________________________________________________________________________
dense_4 (Dense)                 (None, 4)            52          concatenate_1[0][0]              
__________________________________________________________________________________________________
dense_5 (Dense)                 (None, 1)            5           dense_4[0][0]                    
==================================================================================================
Total params: 21,271,189
Trainable params: 21,216,661
Non-trainable params: 54,528
__________________________________________________________________________________________________

I decided to augment the images due to imbalance. I used the following ImageDataGenerator:

aug = ImageDataGenerator(rescale=1/255.,
                        rotation_range=180,
                        height_shift_range=0.2,
                        width_shift_range=0.2,
                        brightness_range=[0.5,1.5],
                        channel_shift_range=100.0,
                        horizontal_flip=True,
                        vertical_flip=True,
                        shear_range=45.0)

I then compiled and attempted to train the model using ImageDataGenerator().flow():

epochs = 10
BATCH_SIZE = 20
flow = aug.flow(img_train, y_train, batch_size=BATCH_SIZE)

history = model.fit([meta_train, flow], y_train, epochs=epochs, batch_size=100, validation_data=([meta_test, img_test], y_test), class_weight=class_weight)

This gives me an error:

ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of
types {"<class 'pandas.core.frame.DataFrame'>", "<class 'tensorflow.python.keras.preprocessing.image.NumpyArrayIterator'>"}),
<class 'numpy.ndarray'>

I've tried multiple versions of the code, but I'm just not familiar enough with the backend to correctly diagnose the problem. Can anyone help me with this?


Model code and MRE

Model code

LEARNING_RATE = 0.001

# Define inputs
meta_inputs = Input(shape=(10,))
img_inputs = Input(shape=(80,120,3,))

# Model 1
meta_layer1 = Dense(4, activation='relu')(meta_inputs)

# Model 2
xception_layer = Xception(include_top=False, input_shape=(80,120,3,))(img_inputs)
img_conv_layer1 = Conv2D(8, kernel_size=(5,5), padding='same', activation='relu')(xception_layer)
img_gap_layer = GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4, activation='relu')(img_gap_layer)

# Merge models
merged_layer = Concatenate()([meta_layer1, img_gap_layer])
merged_dense_layer = Dense(4, activation='relu')(merged_layer)
merged_output = Dense(1, activation='sigmoid')(merged_dense_layer)


# Define functional model
model = Model(inputs=[meta_inputs, img_inputs], outputs=merged_output)

# Compile model
auc = AUC(name = 'auc')
model.compile(Adam(learning_rate=LEARNING_RATE), loss='binary_crossentropy', metrics=[auc])
model.summary()

meta_train MRE

       age_approx  Unknown  female  male  head/neck  lower extremity  \
11655          45        0       0     1          0                0   
24502          60        0       0     1          0                1   
2524           50        0       1     0          0                1   
13894          60        0       1     0          0                0   
29325          45        0       1     0          0                1   

       oral/genital  palms/soles  torso  upper extremity  
11655             0            0      1                0  
24502             0            0      0                0  
2524              0            0      0                0  
13894             0            0      1                0  
29325             0            0      0                0 

img_train MRE

Array too large, see code here.

y_train.shape

(23188, 1)


from Image augmentation in Keras mixed functional model

No comments:

Post a Comment