Keras you are trying to load a weight file containing 2 layers into a model with 1 layers

I have train a CNN model using Keras and store the weights. When I am trying to load them back to the same model I am receiving the following error:

ValueError: You are trying to load a weight file containing 2 layers into a model with 1 layers.

I figure out that this is a common error. However, the proposed remedies did not seem to work for me. I tried to downgrade my current Keras version of 2.2.4 to 2.1.6. My model looks like:

def build_model(self):

    model = Sequential()
    #pdb.set_trace()
    model.add(Dense(128 * 7 * 7, activation="relu", input_shape=(None, self.latent_dim))) #input_dim
    model.add(Reshape((7, 7, 128)))
    model.add(UpSampling2D())
    model.add(Conv2D(128, kernel_size=4, padding="same"))
    model.add(BatchNormalization(momentum=0.8))
    model.add(Activation("relu"))
    model.add(UpSampling2D())
    model.add(Conv2D(64, kernel_size=4, padding="same"))
    model.add(BatchNormalization(momentum=0.8))
    model.add(Activation("relu"))

    model.add(UpSampling2D(size=(1, 2)))
    model.add(Conv2D(64, kernel_size=4, padding="same"))
    model.add(BatchNormalization(momentum=0.8))
    model.add(Activation("relu"))

    model.add(UpSampling2D(size=(1, 2)))
    model.add(Conv2D(64, kernel_size=4, padding="same"))
    model.add(BatchNormalization(momentum=0.8))
    model.add(Activation("relu"))

    model.add(Conv2D(self.channels, kernel_size=4, padding="same"))
    model.add(Activation("tanh"))

    model.summary()

    noise = Input(shape=(self.latent_dim,))
    img = model(noise)

    return Model(noise, img)

Then, for loading the weights I am doing something like:

self.my_model = self.build_model()
self.my_model.load_weights('models/stored_weights') 

Why am I receiving that message? And what can I do to solve it?

from Keras you are trying to load a weight file containing 2 layers into a model with 1 layers