Wednesday, 12 May 2021

Neural network with linear activation output. Calculate output range for each of the output neurons

Let's suposse I have a neural network like the following:

model = keras.models.Sequential()
model.add(keras.layers.Dense(10, input_shape=(5,), activation='relu'))
model.add(keras.layers.Dense(4, activation='linear'))

With n output neurons with a linear activation function.

The train process is not important here, so we can take a look to the random weights that keras initialized using:

model.weights 

Of course, in a real example these weights should be adjusted in the training process.

Depending on these model.weights, each of the output neurons returns values in a range.

I would like to calculate this exact range.

Does keras offer any function to calculate it?

I built a flawed piece of code to make an aproximation of it, using a loop and predicting random inputs. But this would not be really usable in a real example with much more inputs/neurons/weights.

Here a few examples trying to clarify my question (All of them assume that the input values are between and 1):

model = keras.models.Sequential()
model.add(keras.layers.Dense(1, input_shape=(2,), activation='linear', use_bias=False))

model.set_weights([np.array([1, 1]).reshape(2, 1)]) 

For the previous example the output neuron results would be between 0 and 2

model.set_weights([np.array([-0.5, 1]).reshape(2, 1)])

For the previous example the output neuron results would be between -0.5 and 1

model = keras.models.Sequential()
model.add(keras.layers.Dense(2, input_shape=(2,), activation='linear', use_bias=False))
model.add(keras.layers.Dense(1, activation='linear', use_bias=False))

model.set_weights([np.array([1, 1, 1, 1]).reshape(2,2), np.array([1, 1]).reshape(2,1)])

For the previous example the output neuron results would be between 0 and 4

These are simplified examples. In a real scenario with a much complex network structure, activation functions, bias..... these ranges are not obvious to calculate.



from Neural network with linear activation output. Calculate output range for each of the output neurons

No comments:

Post a Comment