How to set layer-wise learning rate in Tensorflow?

by

It can be achieved quite easily with 2 optimizers:

var_list1 = [variables from first 5 layers]
var_list2 = [the rest of variables]
train_op1 = GradientDescentOptimizer(0.00001).minimize(loss, var_list=var_list1)
train_op2 = GradientDescentOptimizer(0.0001).minimize(loss, var_list=var_list2)
train_op = tf.group(train_op1, train_op2)

One disadvantage of this implementation is that it computes tf.gradients(.) twice inside the optimizers and thus it might not be optimal in terms of execution speed. This can be mitigated by explicitly calling tf.gradients(.), splitting the list into 2 and passing corresponding gradients to both optimizers.

Related question: Holding variables constant during optimizer

EDIT: Added more efficient but longer implementation:

var_list1 = [variables from first 5 layers]
var_list2 = [the rest of variables]
opt1 = tf.train.GradientDescentOptimizer(0.00001)
opt2 = tf.train.GradientDescentOptimizer(0.0001)
grads = tf.gradients(loss, var_list1 + var_list2)
grads1 = grads[:len(var_list1)]
grads2 = grads[len(var_list1):]
tran_op1 = opt1.apply_gradients(zip(grads1, var_list1))
train_op2 = opt2.apply_gradients(zip(grads2, var_list2))
train_op = tf.group(train_op1, train_op2)

You can use tf.trainable_variables() to get all training variables and decide to select from them.
The difference is that in the first implementation tf.gradients(.) is called twice inside the optimizers. This may cause some redundant operations to be executed (e.g. gradients on the first layer can reuse some computations for the gradients of the following layers).

More Related Contents:

  1. Keras, How to get the output of each layer?
  2. How to make a custom activation function with only Python in Tensorflow?
  3. Using a pre-trained word embedding (word2vec or Glove) in TensorFlow
  4. What is the difference between ‘SAME’ and ‘VALID’ padding in tf.nn.max_pool of tensorflow?
  5. How to apply gradient clipping in TensorFlow?
  6. How to assign a value to a TensorFlow variable?
  7. Keras replacing input layer
  8. Keras custom loss function: Accessing current input pattern
  9. How can I use a pre-trained neural network with grayscale images?
  10. How to understand the term `tensor` in TensorFlow?
  11. Dimension of shape in conv1D
  12. TensorFlow ValueError: Cannot feed value of shape (64, 64, 3) for Tensor u’Placeholder:0′, which has shape ‘(?, 64, 64, 3)’
  13. Removing then Inserting a New Middle Layer in a Keras Model
  14. Unbalanced data and weighted cross entropy
  15. Custom loss function in Keras
  16. How do I get the weights of a layer in Keras?
  17. Multiple outputs in Keras
  18. How to handle non-determinism when training on a GPU?
  19. ValueError: Input 0 of layer sequential is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [8, 28, 28]
  20. Keras LSTM input dimension setting
  21. How to implement Grad-CAM on a trained network
  22. looking for source code of from gen_nn_ops in tensorflow
  23. ValueError: Tensor must be from the same graph as Tensor with Bidirectinal RNN in Tensorflow
  24. What is right batch normalization function in Tensorflow?
  25. How to fix “ResourceExhaustedError: OOM when allocating tensor”
  26. How do I plot a Keras/Tensorflow subclassing API model?
  27. Strange behaviour of the loss function in keras model, with pretrained convolutional base
  28. Multi-input Multi-output Model with Keras Functional API
  29. How to save/restore a model after training?
  30. Can Keras with Tensorflow backend be forced to use CPU or GPU at will?

Leave a Comment