How do I split Tensorflow datasets?

by

You may use Dataset.take() and Dataset.skip():

train_size = int(0.7 * DATASET_SIZE)
val_size = int(0.15 * DATASET_SIZE)
test_size = int(0.15 * DATASET_SIZE)

full_dataset = tf.data.TFRecordDataset(FLAGS.input_file)
full_dataset = full_dataset.shuffle()
train_dataset = full_dataset.take(train_size)
test_dataset = full_dataset.skip(train_size)
val_dataset = test_dataset.skip(test_size)
test_dataset = test_dataset.take(test_size)

For more generality, I gave an example using a 70/15/15 train/val/test split but if you don’t need a test or a val set, just ignore the last 2 lines.

Take:

Creates a Dataset with at most count elements from this dataset.

Skip:

Creates a Dataset that skips count elements from this dataset.

You may also want to look into Dataset.shard():

Creates a Dataset that includes only 1/num_shards of this dataset.

More Related Contents:

  1. Meaning of buffer_size in Dataset.map , Dataset.prefetch and Dataset.shuffle
  2. Split a dataset created by Tensorflow dataset API in to Train and Test?
  3. How to extract data/labels back from TensorFlow dataset
  4. tf.data with multiple inputs / outputs in Keras
  5. parallelising tf.data.Dataset.from_generator
  6. Parallelism isn’t reducing the time in dataset map
  7. Which TensorFlow and CUDA version combinations are compatible?
  8. Can I measure the execution time of individual operations with TensorFlow?
  9. How to import an saved Tensorflow model train using tf.estimator and predict on input data
  10. Keras : How should I prepare input data for RNN?
  11. What’s the difference of name scope and a variable scope in tensorflow?
  12. How does TensorFlow name tensors?
  13. What is the meaning of the “None” in model.summary of KERAS?
  14. Module ‘tensorflow’ has no attribute ‘contrib’
  15. Understanding TensorBoard (weight) histograms
  16. Confused by the behavior of `tf.cond`
  17. TensorFlow: slow performance when getting gradients at inputs
  18. TensorFlow, why there are 3 files after saving the model?
  19. tensorflow on GPU: no known devices, despite cuda’s deviceQuery returning a “PASS” result
  20. How can I run Tensorflow on one single core?
  21. Holding variables constant during optimizer
  22. Is there a simpler way to handle batch inputs from tfrecords?
  23. Obtaining output of an Intermediate layer in TensorFlow/Keras
  24. ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory
  25. Could not install packages due to an EnvironmentError: [WinError 5] Access is denied:
  26. Convert between NHWC and NCHW in TensorFlow
  27. How to properly use tf.metrics.accuracy?
  28. How can I convert a trained Tensorflow model to Keras?
  29. ImportError: libcudnn when running a TensorFlow program
  30. Google Storage (gs) wrapper file input/out for Cloud ML?

Leave a Comment