Keras misinterprets training data shape

by

(Edited, according to OP’s comment on this question, where they posted this link: https://github.com/fchollet/keras/issues/1920)

Your X is not a single numpy array, it’s an array of arrays. (Otherwise its shape would be X.shape=(35730,513,15).

It must be a single numpy array for the fit method. Since you have a variable length, you cannot have a single numpy array containing all your data, you will have to divide it in smaller arrays, each array containing data with the same length.

For that, you should maybe create a dictionary by shape, and loop the dictionary manually (there may be other better ways to do this…):

#code in python 3.5
xByShapes = {}
yByShapes = {}
for itemX,itemY in zip(X,Y):
    if itemX.shape in xByShapes:
        xByShapes[itemX.shape].append(itemX)
        yByShapes[itemX.shape].append(itemY)
    else:
        xByShapes[itemX.shape] = [itemX] #initially a list, because we're going to append items
        yByShapes[itemX.shape] = [itemY]

At the end, you loop this dictionary for training:

for shape in xByShapes:
    model.fit(
              np.asarray(xByShapes[shape]), 
              np.asarray(yByShapes[shape]),...
              )

Masking

Alternatively, you can pad your data so all samples have the same length, using zeros or some dummy value.

Then before anything in your model you can add a Masking layer that will ignore these padded segments. (Warning: some types of layer don’t support masking)

More Related Contents:

  1. TensorFlow Only running on 1/32 of the Training data provided [duplicate]
  2. Make a custom loss function in keras
  3. Keras not training on entire dataset
  4. How to export Keras .h5 to tensorflow .pb?
  5. Could not load dynamic library ‘cudart64_101.dll’ on tensorflow CPU-only installation
  6. How to apply gradient clipping in TensorFlow?
  7. Keras + Tensorflow and Multiprocessing in Python
  8. NotImplementedError: Layers with arguments in `__init__` must override `get_config`
  9. Can Keras with Tensorflow backend be forced to use CPU or GPU at will?
  10. Keras replacing input layer
  11. How does Keras 1d convolution layer work with word embeddings – text classification problem? (Filters, kernel size, and all hyperparameter)
  12. Keras Sequential model with multiple inputs
  13. ValueError at /image/ Tensor Tensor(“activation_5/Softmax:0”, shape=(?, 4), dtype=float32) is not an element of this graph
  14. What do I need K.clear_session() and del model for (Keras with Tensorflow-gpu)?
  15. expected ndim=3, found ndim=2
  16. This model has not yet been built error on model.summary()
  17. How to get other metrics in Tensorflow 2.0 (not only accuracy)?
  18. How can I use a pre-trained neural network with grayscale images?
  19. Removing then Inserting a New Middle Layer in a Keras Model
  20. Why can’t I get reproducible results in Keras even though I set the random seeds?
  21. AttributeError: module ‘tensorflow.python.keras.utils.generic_utils’ has no attribute ‘populate_dict_with_module_objects’
  22. Custom loss function in Keras
  23. Save and load model optimizer state
  24. How do I get the weights of a layer in Keras?
  25. ValueError: Input 0 of layer sequential is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [8, 28, 28]
  26. Keras LSTM input dimension setting
  27. ‘Tensor’ object has no attribute ‘lower’
  28. Unexpected keyword argument ‘ragged’ in Keras
  29. ValueError: Output tensors to a Model must be the output of a TensorFlow `Layer`
  30. Invalid Argument Error / Graph Execution Error

Leave a Comment