UPDATE 8/1730/2020: TF 2.3 has finally done it: all cases run as fast, or notably faster, than any previous version. Further, my previous update was unfair to TF; my GPU was to blame, has been overheating lately. If you see a rising stem plot of iteration times, it’s a reliable symptom. Lastly, see a dev’s … Read more
From my experience – the problem lies in loading Keras to one process and then spawning a new process when the keras has been loaded to your main environment. But for some applications (like e.g. training a mixture of Kerasmodels) it’s simply better to have all of this things in one process. So what I … Read more
Gradient clipping needs to happen after computing the gradients, but before applying them to update the model’s parameters. In your example, both of those things are handled by the AdamOptimizer.minimize() method. In order to clip your gradients you’ll need to explicitly compute, clip, and apply them as described in this section in TensorFlow’s API documentation. … Read more
I’ve seen this error message for three different reasons, with different solutions: 1. You have cache issues I regularly work around this error by shutting down my python process, removing the ~/.nv directory (on linux, rm -rf ~/.nv), and restarting the Python process. I don’t exactly know why this works. It’s probably at least partly … Read more
There are lots of things I have seen make a model diverge. Too high of a learning rate. You can often tell if this is the case if the loss begins to increase and then diverges to infinity. I am not to familiar with the DNNClassifier but I am guessing it uses the categorical cross … Read more
If you only want to predict the output using the most recent 5 inputs, there is no need to ever provide the full 600 time steps of any training sample. My suggestion would be to pass the training data in the following manner: t=0 t=1 t=2 t=3 t=4 t=5 … t=598 t=599 sample0 |———————| sample0 … Read more
Credits to this Github issue comment by Ritchie Ng. # Creating a model from keras.models import Sequential from keras.layers import Dense # Custom activation function from keras.layers import Activation from keras import backend as K from keras.utils.generic_utils import get_custom_objects def custom_activation(x): return (K.sigmoid(x) * 5) – 1 get_custom_objects().update({‘custom_activation’: Activation(custom_activation)}) # Usage model = Sequential() model.add(Dense(32, … Read more
TL;DR Several possible errors, most fixed with x = np.asarray(x).astype(‘float32′). Others may be faulty data preprocessing; ensure everything is properly formatted (categoricals, nans, strings, etc). Below shows what the model expects: [print(i.shape, i.dtype) for i in model.inputs] [print(o.shape, o.dtype) for o in model.outputs] [print(l.name, l.input_shape, l.dtype) for l in model.layers] The problem’s rooted in using … Read more
Tensorflow 2.1+ What’s going on? With the new Tensorflow 2.1 release, the default tensorflow pip package contains both CPU and GPU versions of TF. In previous TF versions, not finding the CUDA libraries would emit an error and raise an exception, while now the library dynamically searches for the correct CUDA version and, if it … Read more
You can find the answer at the Keras docs: https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development. In short, to be absolutely sure that you will get reproducible results with your python script on one computer’s/laptop’s CPU then you will have to do the following: Set the PYTHONHASHSEED environment variable at a fixed value Set the python built-in pseudo-random generator at a … Read more