How to set adaptive learning rate for GradientDescentOptimizer?

by

First of all, tf.train.GradientDescentOptimizer is designed to use a constant learning rate for all variables in all steps. TensorFlow also provides out-of-the-box adaptive optimizers including the tf.train.AdagradOptimizer and the tf.train.AdamOptimizer, and these can be used as drop-in replacements.

However, if you want to control the learning rate with otherwise-vanilla gradient descent, you can take advantage of the fact that the learning_rate argument to the tf.train.GradientDescentOptimizer constructor can be a Tensor object. This allows you to compute a different value for the learning rate in each step, for example:

learning_rate = tf.placeholder(tf.float32, shape=[])
# ...
train_step = tf.train.GradientDescentOptimizer(
    learning_rate=learning_rate).minimize(mse)

sess = tf.Session()

# Feed different values for learning rate to each training step.
sess.run(train_step, feed_dict={learning_rate: 0.1})
sess.run(train_step, feed_dict={learning_rate: 0.1})
sess.run(train_step, feed_dict={learning_rate: 0.01})
sess.run(train_step, feed_dict={learning_rate: 0.01})

Alternatively, you could create a scalar tf.Variable that holds the learning rate, and assign it each time you want to change the learning rate.

More Related Contents:

  1. What are logits? What is the difference between softmax and softmax_cross_entropy_with_logits?
  2. TensorFlow Only running on 1/32 of the Training data provided [duplicate]
  3. How to get current available GPUs in tensorflow?
  4. How to export Keras .h5 to tensorflow .pb?
  5. How to choose cross-entropy loss in TensorFlow?
  6. Keras + Tensorflow and Multiprocessing in Python
  7. Is it still necessary to install CUDA before using the conda tensorflow-gpu package?
  8. Error running basic tensorflow example
  9. Is Tensorflow compatible with a Windows workflow?
  10. Tensorflow, best way to save state in RNNs?
  11. Keras Sequential model with multiple inputs
  12. ValueError at /image/ Tensor Tensor(“activation_5/Softmax:0”, shape=(?, 4), dtype=float32) is not an element of this graph
  13. Streaming large training and test files into Tensorflow’s DNNClassifier
  14. TensorFlow operator overloading
  15. expected ndim=3, found ndim=2
  16. Clearing Tensorflow GPU memory after model execution
  17. Why can’t I get reproducible results in Keras even though I set the random seeds?
  18. NotImplementedError: Cannot convert a symbolic Tensor (lstm_2/strided_slice:0) to a numpy array. T
  19. How convert a jpeg image into json file in Google machine learning
  20. AttributeError: module ‘tensorflow.python.keras.utils.generic_utils’ has no attribute ‘populate_dict_with_module_objects’
  21. How to get stable results with TensorFlow, setting random seed
  22. Unbalanced data and weighted cross entropy
  23. “freeze” some variables/scopes in tensorflow: stop_gradient vs passing variables to minimize
  24. How can I visualize the weights(variables) in cnn in Tensorflow?
  25. Tensor is not an element of this graph
  26. overcome Graphdef cannot be larger than 2GB in tensorflow
  27. Training a simple model in Tensorflow GPU slower than CPU
  28. cannot import name ‘_registerMatType’ from ‘cv2.cv2’
  29. TensorFlow: Non-repeatable results
  30. How to do Xavier initialization on TensorFlow

Leave a Comment