3

I am a beginner with rnns, consider this sample code 

from tensorflow import keras
import numpy as np

if __name__ == '__main__':
    model = keras.Sequential((
        keras.layers.SimpleRNN(5, activation="softmax", input_shape=(1, 3)),
    ))
    X = [
        [1, 2, 3],
        [4, 5, 6]
    ]
    y = [
        [1, 0, 0, 0, 0],
        [0, 1, 0, 0, 0]
    ]
    X = np.array(X)
    X = np.reshape(X, (2, 1, 3))
    y = np.array(y)
    # print(X)
    # print(y)
    print(model.summary())
    model.compile(optimizer=keras.optimizers.Adam(learning_rate=0.01),
                 loss="categorical_crossentropy")
    model.fit(X, y, epochs=100)
    p = model.predict(X)
    print(p)
    p = list(np.squeeze(p))
    print(p)
    print(np.sum(p,axis=1))

I am using a simple rnn with batch size=2, 3 input features and 1 timestep,as the activation is softmax the last line prints [1,1] as the sum of predictions of a softmax is 1. But when when I change the layer from a SimpleRNN to 

keras.layers.LSTM(5, activation="softmax", input_shape= 
                 (1,3),recurrent_activation="softmax")

The sum of predictions is no longer 1, why is that?

user329387
  • 31
  • 1

1 Answers1

1

Softmax doesnt work as an LSTM activation. You have to add a dense layer using a softmax activation after the LSTM layer. I wouls suggest using another activation in the LSTM like relu.

user85857
  • 11
  • 1