How to apply gradient clipping in TensorFlow?

by

Gradient clipping needs to happen after computing the gradients, but before applying them to update the model’s parameters. In your example, both of those things are handled by the AdamOptimizer.minimize() method.

In order to clip your gradients you’ll need to explicitly compute, clip, and apply them as described in this section in TensorFlow’s API documentation. Specifically you’ll need to substitute the call to the minimize() method with something like the following:

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
gvs = optimizer.compute_gradients(cost)
capped_gvs = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gvs]
train_op = optimizer.apply_gradients(capped_gvs)

More Related Contents:

  1. Keras custom loss function: Accessing current input pattern
  2. How can I use a pre-trained neural network with grayscale images?
  3. Removing then Inserting a New Middle Layer in a Keras Model
  4. How to fix “ResourceExhaustedError: OOM when allocating tensor”
  5. Keras, How to get the output of each layer?
  6. Keras Dense layer’s input is not flattened
  7. Make a custom loss function in keras
  8. Deep-Learning Nan loss reasons
  9. Keras Sequential model input layer
  10. Proper way to feed time-series data to stateful LSTM?
  11. Does Any one got “AttributeError: ‘str’ object has no attribute ‘decode’ ” , while Loading a Keras Saved Model
  12. Keras replacing input layer
  13. ValueError at /image/ Tensor Tensor(“activation_5/Softmax:0”, shape=(?, 4), dtype=float32) is not an element of this graph
  14. How to get other metrics in Tensorflow 2.0 (not only accuracy)?
  15. How to understand the term `tensor` in TensorFlow?
  16. Unbalanced data and weighted cross entropy
  17. Custom loss function in Keras
  18. Save and load model optimizer state
  19. What is the role of TimeDistributed layer in Keras?
  20. Unknown initializer: GlorotUniform when loading Keras model
  21. How do I get the weights of a layer in Keras?
  22. Multiple outputs in Keras
  23. Keras LSTM input dimension setting
  24. How to create a keras layer with a custom gradient in TF2.0?
  25. How to implement Grad-CAM on a trained network
  26. keras: how to save the training history attribute of the history object
  27. How do I plot a Keras/Tensorflow subclassing API model?
  28. Strange behaviour of the loss function in keras model, with pretrained convolutional base
  29. How is the Keras Conv1D input specified? I seem to be lacking a dimension
  30. Multi-input Multi-output Model with Keras Functional API

Leave a Comment