Thursday 11 March 2021

Keras Dense Neural Net Predicting in Narrow Range

I've been playing with Numer.ai data, mostly as a way to improve my understanding of neural nets but I'm running into a problem that I can't seem to get past. No matter the configuration of my dense neural net, the output comes out in a tight range. The input is 300 scaled feature columns (0 to 1) and the target is between 0 and 1 (values of 0, 0.25, 0.5, 0.75, and 1)

Here is my fully reproducible code:

import pandas as pd
# load data
training_data = pd.read_csv("https://numerai-public-datasets.s3-us-west-2.amazonaws.com/latest_numerai_training_data.csv.xz")
tournament_data = pd.read_csv("https://numerai-public-datasets.s3-us-west-2.amazonaws.com/latest_numerai_tournament_data.csv.xz")

feature_cols = training_data.columns[training_data.columns.str.startswith('feature')]
# select those columns out of the training dataset
X_train = training_data[feature_cols].to_numpy()
# select target variables
y_train = training_data.loc[:,'target'].to_numpy()

#same thing on validation data
val_data = tournament_data[tournament_data.data_type=='validation']
X_val = val_data[feature_cols]
y_val= val_data.loc[:,'target']

I've tried a number of different configurations in my neural network (different optimizers: adam and sgd, different learning rates 0.01 down to 0.0001, different neuron sizes, adding dropout: although, I didn't expect this to work because it seems to be a problem with bias, not variance, using linear, softmax, and sigmoid final layer activation functions: softmax produces negative values so that was an immediate non-starter, different batch sizes: as small as 16 and as large as 256, adding or removing batch normalization, shuffling the input data, and training for different numbers of epochs). Ultimately, the results are one of two things:

  1. Predicted values are all the same number, usually somewhere in the 0.45 to 0.55 area
  2. Predicted values are in a very narrow range, usually not more than 0.05 different. So the values are 0.45 to 0.55

I can't figure out what configuration changes I need to make to get this neural network to output predictions across a broader area of the 0 to 1 range.

from tensorflow.keras import models, layers

dropout_rate = 0.15

model = models.Sequential()
model.add(layers.Dense(512, input_shape=(X_train.shape[1],)))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(dropout_rate))

model.add(layers.Dense(1028, activation = 'relu', kernel_regularizer='l2'))
model.add(layers.BatchNormalization())
model.add(layers.Dropout(dropout_rate))

model.add(layers.Dense(1, activation='sigmoid'))

model.compile(optimizer='adam',
  loss='binary_crossentropy',metrics=['mae', 'mse'])

history = model.fit(X_train, y_train,
          validation_data=(X_val, y_val),
          batch_size=64,
          epochs=200,
          verbose=1)

# Prediction output
predictions_df = model.predict(X_val)
predictions_df = predictions_df.reshape(len(predictions_df))

pred_max = predictions_df.max()
pred_min = predictions_df.min()
pred_range = pred_max - pred_min

print(pred_max, pred_min, pred_range)
# example output: 0.51895267 0.47968164 0.039271027


from Keras Dense Neural Net Predicting in Narrow Range

No comments:

Post a Comment