Multivariate LSTM with missing values

by

As suggested by François Chollet (creator of Keras) in his book, one way to handle missing values is to replace them with zero:

In general, with neural networks, it’s safe to input missing values as
0, with the condition that 0 isn’t already a meaningful value. The
network will learn from exposure to the data that the value 0 means
missing data and will start ignoring the value. Note that if you’re
expecting missing values in the test data, but the network was trained
on data without any missing values, the network won’t have learned to
ignore missing values! In this situation, you should artificially
generate training samples with missing entries: copy some training
samples several times, and drop some of the features that you expect
are likely to be missing in the test data.

So you can assign zero to NaN elements, considering that zero is not used in your data (you can normalize the data to a range, say [1,2], and then assign zero to NaN elements; or alternatively, you can normalize all the values to be in range [0,1] and then use -1 instead of zero to replace NaN elements.)

Another alternative way is to use a Masking layer in Keras. You give it a mask value, say 0, and it would drop any timestep (i.e. row) where all its features are equal to the mask value. However, all the following layers should support masking and you also need to pre-process your data and assign the mask value to all the features of a timestep which includes one or more NaN features. Example from Keras doc:

Consider a Numpy data array x of shape (samples, timesteps,features),
to be fed to an LSTM layer. You want to mask timestep #3
and #5 because you lack data for these timesteps. You can:

  • set x[:, 3, :] = 0. and x[:, 5, :] = 0.

  • insert a Masking layer with mask_value=0. before the LSTM layer:

model = Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(LSTM(32))

Update (May 2021): According to an updated suggestion from François Cholle, it might be better to use a more meaningful or informative value (instead of using zero) for masking missing values. This value could be computed (e.g. mean, median, etc.) or predicted from the data itself.

More Related Contents:

  1. In Keras, what exactly am I configuring when I create a stateful `LSTM` layer with N `units`?
  2. Keras : How should I prepare input data for RNN?
  3. Choosing number of Steps per Epoch
  4. TensorFlow and person recognition in video stream
  5. Tensorflow – ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type float)
  6. Keras + Tensorflow and Multiprocessing in Python
  7. What is the meaning of the word logits in TensorFlow? [duplicate]
  8. Proper way to feed time-series data to stateful LSTM?
  9. What is the meaning of the “None” in model.summary of KERAS?
  10. Specify connections in NN (in keras)
  11. What are the differences between all these cross-entropy losses in Keras and TensorFlow?
  12. Negative dimension size caused by subtracting 3 from 1 for ‘conv2d_2/convolution’
  13. expected ndim=3, found ndim=2
  14. Passing output of a CNN to BILSTM
  15. Does model.compile() initialize all the weights and biases in Keras (tensorflow backend)?
  16. Keras Loss Function with Additional Dynamic Parameter
  17. Create keras callback to save model predictions and targets for each batch during training
  18. How do I check if keras is using gpu version of tensorflow?
  19. Obtaining output of an Intermediate layer in TensorFlow/Keras
  20. tf.data with multiple inputs / outputs in Keras
  21. ValueError: Shapes (None, 1) and (None, 2) are incompatible
  22. Multiple outputs in Keras
  23. Keras LSTM input dimension setting
  24. Keras split train test set when using ImageDataGenerator
  25. Confusion about keras Model: __call__ vs. call vs. predict methods
  26. How can I convert a trained Tensorflow model to Keras?
  27. How to display custom images in TensorBoard using Keras?
  28. Using pre-trained word2vec with LSTM for word generation
  29. What is the difference between keras and tf.keras?
  30. How to replace (or insert) intermediate layer in Keras model?

Leave a Comment