loss not dropping with pretrained resnet for semantic segmentation

I am trying to apply semantic segmentation.

I am loading pretrained weights and adding some additional layers.

def resnet50(train_generator, val_generator):
    # Load pretrained weights
    imagenet_weights = 'resnet50_weights.h5'
    base_resnet = ResNet50(include_top=False, weights=imagenet_weights, input_shape=(128, 128, 3))

    last_layer = base_resnet.output
    x = Conv2D(512, 3 , activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(last_layer)
    x = Dropout(0.4)(x)
    x = Conv2D(256, 3 , activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x = Dropout(0.4)(x)
    x = Conv2D(256, 3 , activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(x)
    x = Dropout(0.4)(x)

    ch, cw = get_crop_shape(base_resnet.input, x)

    x = ZeroPadding2D(padding=((ch[0], ch[1]), (cw[0], cw[1])))(x)
    output = Conv2D(3, (1, 1), padding='same', activation='sigmoid')(x)

    model = Model(base_resnet.input, output)
    model_checkpoint = ModelCheckpoint('RESNET50.model', verbose=1, save_best_only=True)

    base_resnet.trainable = False
    sgd = SGD(lr=0.0001)

    model_resnet.compile(loss='binary_crossentropy', optimizer=sgd, metrics=['accuracy'])

    history = model.fit_generator(train_generator,
                                         validation_data=val_generator,
                                         steps_per_epoch=steps_per_epoch,
                                         validation_steps=val_steps,
                                         epochs = 40,
                                         shuffle=True,
                                         callbacks=[model_checkpoint])

    return history, model


def get_crop_shape(target, refer):
    # width, the 3rd dimension
    cw = (target.get_shape()[2] - refer.get_shape()[2]).value
    assert (cw >= 0)
    if cw % 2 != 0:
        cw1, cw2 = int(cw/2), int(cw/2) + 1
    else:
        cw1, cw2 = int(cw/2), int(cw/2)
    # height, the 2nd dimension
    ch = (target.get_shape()[1] - refer.get_shape()[1]).value
    assert (ch >= 0)
    if ch % 2 != 0:
        ch1, ch2 = int(ch/2), int(ch/2) + 1
    else:
        ch1, ch2 = int(ch/2), int(ch/2)

    return (ch1, ch2), (cw1, cw2)

The problem is that the validation loss and training loss are almost not dropping (values start from 0.68 and after 40 epochs go to 0.60).

The accuracy is almost steady around 0.75.

The model is not complex enough?I have tried some combinations but still getting the same results.

Do, I have to implement something like the unet ? After loading the weights?

I tried that also but I am getting some errors using Maxpooling.And I am not sure if this is right.

I have normalized data by dividing with 255, used different optimizers (adam , rmsprop) and learning rates from 0.1 to 0.00001.

from loss not dropping with pretrained resnet for semantic segmentation