Using the Transformer architecture from Attention Is All You Need and it's implementation from Transformer model for language understanding, I want to change the model to accept as input my one dimensional feature array using a Dense layer with 30 units (for encoder), and classify into 6 classes as onehot using Dense with 6 units (for decoder).
The first thing I tried to change is Encoder class :
class Encoder(tf.keras.layers.Layer):
def __init__(self,*, num_layers, d_model, num_heads, dff, input_size,
rate=0.1):
super(Encoder, self).__init__()
self.d_model = d_model
self.num_layers = num_layers
# self.input_layer = tf.keras.layers.Input(shape=(None, input_size))
self.first_hidden_layer = tf.keras.layers.Dense(d_model, activation='relu', input_shape=(input_size,))
self.pos_encoding = positional_encoding(input_size, self.d_model)
self.enc_layers = [
EncoderLayer(d_model=d_model, num_heads=num_heads, dff=dff, rate=rate)
for _ in range(num_layers)]
self.dropout = tf.keras.layers.Dropout(rate)
def call(self, x, training, mask):
seq_len = tf.shape(x)[1]
x = tf.reshape(x, [64, 29])
# x = self.input_layer(x)
x = self.first_hidden_layer(x)
x *= tf.math.sqrt(tf.cast(self.d_model, tf.float32))
x += self.pos_encoding[:, :seq_len, :]
x = self.dropout(x, training=training)
for i in range(self.num_layers):
x = self.enc_layers[i](x, training, mask)
return x # (batch_size, input_seq_len, d_model)
then
transformer = Transformer(num_layers=num_layers,d_model=d_model,num_heads=num_heads,dff=dff,input_size=input_size,target_size=target_size,rate=dropout_rate)
but I get many errors on different attempts to add a dense layer to the encoder's input:
ValueError: The last dimension of the inputs to `Dense` should be defined. Found `None`
ValueError: slice index 1 of dimension 0 out of bounds.
'Tensor' object is not callable
I know this model is adapted for natural language task like translating, but my current level of knowledge is very low on this task and it will take a lot of time to learn every details. I just need to test one of my assumption quickly to move forward with others. If you know how to adapt this model with my custom input shape (30, ) and output shape (6, ) I would appreciate a lot.
from Custom input and output for transformer model
No comments:
Post a Comment