TensorFlow: Remember LSTM state for next batch (stateful LSTM)

by

I found out it was easiest to save the whole state for all layers in a placeholder.

init_state = np.zeros((num_layers, 2, batch_size, state_size))

...

state_placeholder = tf.placeholder(tf.float32, [num_layers, 2, batch_size, state_size])

Then unpack it and create a tuple of LSTMStateTuples before using the native tensorflow RNN Api.

l = tf.unpack(state_placeholder, axis=0)
rnn_tuple_state = tuple(
[tf.nn.rnn_cell.LSTMStateTuple(l[idx][0], l[idx][1])
 for idx in range(num_layers)]
)

RNN passes in the API:

cell = tf.nn.rnn_cell.LSTMCell(state_size, state_is_tuple=True)
cell = tf.nn.rnn_cell.MultiRNNCell([cell]*num_layers, state_is_tuple=True)
outputs, state = tf.nn.dynamic_rnn(cell, x_input_batch, initial_state=rnn_tuple_state)

The state – variable will then be feeded to the next batch as a placeholder.

More Related Contents:

  1. Tensorflow – ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float)
  2. Proper way to feed time-series data to stateful LSTM?
  3. expected ndim=3, found ndim=2
  4. Passing output of a CNN to BILSTM
  5. Error when checking model input: expected lstm_1_input to have 3 dimensions, but got array with shape (339732, 29)
  6. Keras LSTM input dimension setting
  7. ValueError: Input 0 is incompatible with layer lstm_13: expected ndim=3, found ndim=4
  8. ValueError: Tensor must be from the same graph as Tensor with Bidirectinal RNN in Tensorflow
  9. How to visualize RNN/LSTM gradients in Keras/TensorFlow?
  10. Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
  11. How to tell if tensorflow is using gpu acceleration from inside python shell?
  12. Using a pre-trained word embedding (word2vec or Glove) in TensorFlow
  13. What is the difference between ‘SAME’ and ‘VALID’ padding in tf.nn.max_pool of tensorflow?
  14. Tensorflow read images with labels
  15. TensorFlow version 1.0.0-rc2 on Windows: “OpKernel (‘op: “BestSplits” device_type: “CPU”‘) for unknown op: BestSplits” with test code
  16. Which seeds have to be set where to realize 100% reproducibility of training results in tensorflow?
  17. Difference between Variable and get_variable in TensorFlow
  18. How to use a Keras RNN model to forecast for future dates or events?
  19. Processing time gets longer and longer after each iteration (TensorFlow)
  20. List of tensor names in graph in Tensorflow
  21. how to install tensorflow on anaconda python 3.6
  22. Illegal instruction (core dumped) after running import tensorflow
  23. error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support
  24. How do I check if keras is using gpu version of tensorflow?
  25. How to set layer-wise learning rate in Tensorflow?
  26. Tensorflow : logits and labels must have the same first dimension
  27. Cartesian Product in Tensorflow
  28. How to fix “AttributeError: module ‘tensorflow’ has no attribute ‘get_default_graph'”?
  29. What is right batch normalization function in Tensorflow?
  30. Strange behaviour of the loss function in keras model, with pretrained convolutional base

Leave a Comment