Monday, 29 October 2018

Keras Matrix Multiplication to get Predicted Values

I'm looking to take the output of a Keras model to manually calculate the predicted values through matrix multiplication. I would like to do this to help understand how Keras is working under the hood. I'll use the simple XOR problem. Here is my code:

import numpy as np
import keras
from keras.models import Sequential
from keras.layers.core import Dense
from keras.callbacks import LambdaCallback

class LossHistory(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.losses = []

    def on_batch_end(self, batch, logs={}):
        self.losses.append(logs.get('loss'))


history = LossHistory()

# the four different states of the XOR gate
training_data = np.array([[0,0],[0,1],[1,0],[1,1]], "float32")

# the four expected results in the same order
target_data = np.array([[0],[1],[1],[0]], "float32")

model = Sequential()
model.add(Dense(4, input_dim=2, activation='relu'))
model.add(Dense(1, activation='sigmoid'))

print_weights = LambdaCallback(on_epoch_end=lambda batch, logs: print(model.layers[0].get_weights()))

model.compile(loss='mean_squared_error',
              optimizer='adam',
              metrics=['binary_accuracy'])

history2 = model.fit(training_data, target_data, epochs=50, verbose=2, callbacks=[print_weights, history])

print(model.predict(training_data).round())


W1 = model.get_weights()[0]
X1 = np.matrix([[0,0],[1,1]], "float32")
wx = np.dot(X1,W1)
b = model.get_weights()[1]
wx = np.reshape(wx,(4,2))
b = np.reshape(b, (4,1))
z = wx + b
from numpy import array, exp
a1 = 1 / (1 + exp(-z))
print('g =\n', a1)

W2 = model.get_weights()[2]
b2 = model.get_weights()[3]
W2 = np.reshape(W2,(1,4))
a1 = np.reshape(a1, (4,1))
wa = np.dot(W2,a1)
z2 = wa + b2
a2 = 1 / (1 + exp(-z2))
print('g =\n', a2)

From what I understand, get_weights()[0] and get_weights()[1] are the weights and biases for the first layer, respectively, and get_weights()[2] and get_weights()[3] are the weights and biases for the second layer. I believe the issue I'm have is figuring out what x1 and x2 are as they relate to the equation z = Wx + b. The weights are retrieved from the last epoch and are usually weights that achieve 100% accuracy. The output I'm expecting is [0,1,1,0] for the y-hat predictions based on the manual calculation of z = Wx + b and then taking the sigmoid of z.



from Keras Matrix Multiplication to get Predicted Values

No comments:

Post a Comment