How do I set TensorFlow RNN state when state_is_tuple=True?

by

One problem with a Tensorflow placeholder is that you can only feed it with a Python list or Numpy array (I think). So you can’t save the state between runs in tuples of LSTMStateTuple.

I solved this by saving the state in a tensor like this

initial_state = np.zeros((num_layers, 2, batch_size, state_size))

You have two components in an LSTM layer, the cell state and hidden state, thats what the “2” comes from. (this article is great: https://arxiv.org/pdf/1506.00019.pdf)

When building the graph you unpack and create the tuple state like this:

state_placeholder = tf.placeholder(tf.float32, [num_layers, 2, batch_size, state_size])
l = tf.unpack(state_placeholder, axis=0)
rnn_tuple_state = tuple(
         [tf.nn.rnn_cell.LSTMStateTuple(l[idx][0],l[idx][1])
          for idx in range(num_layers)]
)

Then you get the new state the usual way

cell = tf.nn.rnn_cell.LSTMCell(state_size, state_is_tuple=True)
cell = tf.nn.rnn_cell.MultiRNNCell([cell] * num_layers, state_is_tuple=True)

outputs, state = tf.nn.dynamic_rnn(cell, series_batch_input, initial_state=rnn_tuple_state)

It shouldn’t be like this… perhaps they are working on a solution.

More Related Contents:

  1. What are logits? What is the difference between softmax and softmax_cross_entropy_with_logits?
  2. Keras Dense layer’s input is not flattened
  3. Make a custom loss function in keras
  4. Keras not training on entire dataset
  5. Deep-Learning Nan loss reasons
  6. Can Keras with Tensorflow backend be forced to use CPU or GPU at will?
  7. Proper way to feed time-series data to stateful LSTM?
  8. ValueError at /image/ Tensor Tensor(“activation_5/Softmax:0”, shape=(?, 4), dtype=float32) is not an element of this graph
  9. Keras custom loss function: Accessing current input pattern
  10. TensorFlow operator overloading
  11. How to get other metrics in Tensorflow 2.0 (not only accuracy)?
  12. How can I use a pre-trained neural network with grayscale images?
  13. Removing then Inserting a New Middle Layer in a Keras Model
  14. error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Cocoa support
  15. How to know if underfitting or overfitting is occuring?
  16. Unbalanced data and weighted cross entropy
  17. Save and load model optimizer state
  18. Unknown initializer: GlorotUniform when loading Keras model
  19. Tensor is not an element of this graph
  20. How to handle non-determinism when training on a GPU?
  21. How to create a keras layer with a custom gradient in TF2.0?
  22. Tensorflow One Hot Encoder?
  23. Tensorflow Precision / Recall / F1 score and Confusion matrix
  24. ‘Tensor’ object has no attribute ‘lower’
  25. ValueError: Output tensors to a Model must be the output of a TensorFlow `Layer`
  26. How to fix “ResourceExhaustedError: OOM when allocating tensor”
  27. Attach a queue to a numpy array in tensorflow for data fetch instead of files?
  28. How is the Keras Conv1D input specified? I seem to be lacking a dimension
  29. TensorFlow, “‘module’ object has no attribute ‘placeholder'”
  30. How do you read Tensorboard files programmatically?

Leave a Comment