Custom ImageDataGenerator() for half a million images where labels and pixels are in 2 separate DataFrames using Keras (or any other library) [closed]

by

This is what i would suggest, load the dataframe, piece by piece, do not load the entirety of it at the same time, this might actually exceed your RAM, hence the dying kernel.

Then iterate through the dataframe line by line, take the 32332 columns, and reshape them into an image of 137×236 and save them to disk with a the appropriate name in to the folder train_data/class_number/, you can then use keras ImageDataGenerator().flowfromDirectory()

the issue is that the 32332 columns dont make sense to me, if the image was a single channel 137×236 image, then the number of columns would be 137*236 = 29972.So theres like 2k columns unaccounted for. Are you sure of the format of the data?

More Related Contents:

  1. How can I use a weight matrix created by a trained neural network to make predictions in another file?
  2. Keras + Tensorflow and Multiprocessing in Python
  3. How does Keras calculate the accuracy?
  4. Keras Sequential model input layer
  5. How to concatenate two layers in keras?
  6. NaN loss when training regression network
  7. Neural Network LSTM input shape from dataframe
  8. Negative dimension size caused by subtracting 3 from 1 for ‘conv2d_2/convolution’
  9. Loading a trained Keras model and continue training
  10. How do I check if keras is using gpu version of tensorflow?
  11. How does Keras handle multilabel classification?
  12. What is the role of TimeDistributed layer in Keras?
  13. How to use fit_generator with multiple inputs
  14. Where do I call the BatchNormalization function in Keras?
  15. Multiple outputs in Keras
  16. keras: how to save the training history attribute of the history object
  17. How to log Keras loss output to a file
  18. Can I send callbacks to a KerasClassifier?
  19. Keras give input to intermediate layer and get final output
  20. how to implement custom metric in keras?
  21. How to provide a reproducible copy of your DataFrame with to_clipboard()
  22. Combine two columns of text in pandas dataframe
  23. Pandas convert dataframe to array of tuples
  24. custom matplotlib plot : chess board like table with colored cells
  25. How do you plot a vertical line on a time series plot in Pandas?
  26. Bin pandas dataframe by every X rows
  27. expected ndim=3, found ndim=2
  28. Installing OpenCV on Windows 7 for Python 2.7
  29. pandas apply function that returns multiple values to rows in pandas dataframe
  30. Improve Pandas Merge performance

Leave a Comment