When does keras reset an LSTM state?

by

Cheking with some tests, I got to the following conclusion, which is according to the documentation and to Nassim’s answer:

First, there isn’t a single state in a layer, but one state per sample in the batch. There are batch_size parallel states in such a layer.

Stateful=False

In a stateful=False case, all the states are resetted together after each batch.

  • A batch with 10 sequences would create 10 states, and all 10 states are resetted automatically after it’s processed.

  • The next batch with 10 sequences will create 10 new states, which will also be resetted after this batch is processed

If all those sequences have length (timesteps) = 7, the practical result of these two batches is:

20 individual sequences, each with length 7

None of the sequences are related. But of course: the weights (not the states) will be unique for the layer, and will represent what the layer has learned from all the sequences.

  • A state is: Where am I now inside a sequence? Which time step is it? How is this particular sequence behaving since its beginning up to now?
  • A weight is: What do I know about the general behavior of all sequences I’ve seen so far?

Stateful=True

In this case, there is also the same number of parallel states, but they will simply not be resetted at all.

  • A batch with 10 sequences will create 10 states that will remain as they are at the end of the batch.

  • The next batch with 10 sequences (it’s required to be 10, since the first was 10) will reuse the same 10 states that were created before.

The practical result is: the 10 sequences in the second batch are just continuing the 10 sequences of the first batch, as if there had been no interruption at all.

If each sequence has length (timesteps) = 7, then the actual meaning is:

10 individual sequences, each with length 14

When you see that you reached the total length of the sequences, then you call model.reset_states(), meaning you will not continue the previous sequences anymore, now you will start feeding new sequences.

More Related Contents:

  1. Understanding Keras LSTMs
  2. Keras Dense layer’s input is not flattened
  3. Keras input explanation: input_shape, units, batch_size, dim, etc
  4. Tensorflow – ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float)
  5. How do you create a custom activation function with Keras?
  6. Keras : How should I prepare input data for RNN?
  7. Proper way to feed time-series data to stateful LSTM?
  8. Custom loss function in Keras based on the input data
  9. How to calculate F1 Macro in Keras?
  10. Multivariate LSTM with missing values
  11. Specify connections in NN (in keras)
  12. Neural Network LSTM input shape from dataframe
  13. Negative dimension size caused by subtracting 3 from 1 for ‘conv2d_2/convolution’
  14. expected ndim=3, found ndim=2
  15. Passing output of a CNN to BILSTM
  16. Error when checking model input: expected lstm_1_input to have 3 dimensions, but got array with shape (339732, 29)
  17. How do I create a variable-length input LSTM in Keras?
  18. How do I get the weights of a layer in Keras?
  19. TimeDistributed(Dense) vs Dense in Keras – Same number of parameters
  20. Keras LSTM input dimension setting
  21. What is the meaning of axis=-1 in keras.argmax?
  22. In Keras, what exactly am I configuring when I create a stateful `LSTM` layer with N `units`?
  23. How to add and remove new layers in keras after loading weights?
  24. Error when checking model input: expected convolution2d_input_1 to have 4 dimensions, but got array with shape (32, 32, 3)
  25. How to fix “AttributeError: module ‘tensorflow’ has no attribute ‘get_default_graph'”?
  26. ValueError: Input 0 is incompatible with layer lstm_13: expected ndim=3, found ndim=4
  27. Using pre-trained word2vec with LSTM for word generation
  28. Keras give input to intermediate layer and get final output
  29. Does Any one got “AttributeError: ‘str’ object has no attribute ‘decode’ ” , while Loading a Keras Saved Model
  30. ValueError at /image/ Tensor Tensor(“activation_5/Softmax:0”, shape=(?, 4), dtype=float32) is not an element of this graph

Leave a Comment