The problem here is that the only operation in the Dataset.map() function is a tf.py_func() op. This op calls back into the local Python interpreter to run a function in the same process. Increasing num_parallel_calls will increase the number of TensorFlow threads that attempt to call back into Python concurrently. However, Python has something called … Read more
Turns out I can use Dataset.map if I make the generator super lightweight (only generating meta data) and then move the actual heavy lighting into a stateless function. This way I can parallelise just the heavy lifting part with .map using a py_func. Works; but feels a tad clumsy… Would be great to be able … Read more
The whole process is simplied using the Dataset API. Here are both the parts: (1): Convert numpy array to tfrecords and (2): read the tfrecords to generate batches. 1. Creation of tfrecords from a numpy array: Example arrays: inputs = np.random.normal(size=(5, 32, 32, 3)) labels = np.random.randint(0,2,size=(5,)) def npy_to_tfrecords(inputs, labels, filename): with tf.io.TFRecordWriter(filename) as writer: … Read more
I’m not using Keras but I would go with an tf.data.Dataset.from_generator() – like: def _input_fn(): sent1 = np.array([1, 2, 3, 4, 5, 6, 7, 8], dtype=np.int64) sent2 = np.array([20, 25, 35, 40, 600, 30, 20, 30], dtype=np.int64) sent1 = np.reshape(sent1, (8, 1, 1)) sent2 = np.reshape(sent2, (8, 1, 1)) labels = np.array([40, 30, 20, 10, … Read more
In tensorflow 2.0.0, this will work: import tensorflow as tf raw_dataset = tf.data.TFRecordDataset(“input_file.tfrecord”) shards = 10 for i in range(shards): writer = tf.data.experimental.TFRecordWriter(f”output_file-part-{i}.tfrecord”) writer.write(raw_dataset.shard(shards, i))
The problem is in your target shape and is related to the correct choice of an appropriate loss function. you have 2 possibilities: 1. possibility: if you have 1D integer encoded target, you can use sparse_categorical_crossentropy as loss function n_class = 3 n_features = 100 n_sample = 1000 X = np.random.randint(0,10, (n_sample,n_features)) y = np.random.randint(0,n_class, … Read more
If you are interested in how to input your own data in TensorFlow, you can look at this tutorial. I’ve also written a guide with best practices for CS230 at Stanford here. New answer (with tf.data) and with labels With the introduction of tf.data in r1.4, we can create a batch of images without placeholders … Read more
In case your tf.data.Dataset is batched, the following code will retrieve all the y labels: y = np.concatenate([y for x, y in ds], axis=0) Quick explanation: [y for x, y in ds] is known as “list comprehension” in python. If dataset is batched, this expression will loop thru each batch and put each batch y … Read more
The whole process is simplied using the Dataset API. Here are both the parts: (1): Convert numpy array to tfrecords and (2): read the tfrecords to generate batches. 1. Creation of tfrecords from a numpy array: Example arrays: inputs = np.random.normal(size=(5, 32, 32, 3)) labels = np.random.randint(0,2,size=(5,)) def npy_to_tfrecords(inputs, labels, filename): with tf.io.TFRecordWriter(filename) as writer: … Read more
Assuming you have all_dataset variable of tf.data.Dataset type: test_dataset = all_dataset.take(1000) train_dataset = all_dataset.skip(1000) Test dataset now has first 1000 elements and the rest goes for training.