Update only part of the word embedding matrix in Tensorflow

by

TL;DR: The default implementation of opt.minimize(loss), TensorFlow will generate a sparse update for word_emb that modifies only the rows of word_emb that participated in the forward pass.

The gradient of the tf.gather(word_emb, indices) op with respect to word_emb is a tf.IndexedSlices object (see the implementation for more details). This object represents a sparse tensor that is zero everywhere, except for the rows selected by indices. A call to opt.minimize(loss) calls AdamOptimizer._apply_sparse(word_emb_grad, word_emb), which makes a call to tf.scatter_sub(word_emb, ...)* that updates only the rows of word_emb that were selected by indices.

If on the other hand you want to modify the tf.IndexedSlices that is returned by opt.compute_gradients(loss, word_emb), you can perform arbitrary TensorFlow operations on its indices and values properties, and create a new tf.IndexedSlices that can be passed to opt.apply_gradients([(word_emb, ...)]). For example, you could cap the gradients using MyCapper() (as in the example) using the following calls:

grad, = opt.compute_gradients(loss, word_emb)
train_op = opt.apply_gradients(
    [tf.IndexedSlices(MyCapper(grad.values), grad.indices)])

Similarly, you could change the set of indices that will be modified by creating a new tf.IndexedSlices with a different indices.

* In general, if you want to update only part of a variable in TensorFlow, you can use the tf.scatter_update(), tf.scatter_add(), or tf.scatter_sub() operators, which respectively set, add to (+=) or subtract from (-=) the value previously stored in a variable.

More Related Contents:

  1. Using Gensim Fasttext model with LSTM nn in keras
  2. Which TensorFlow and CUDA version combinations are compatible?
  3. Can I measure the execution time of individual operations with TensorFlow?
  4. How to import an saved Tensorflow model train using tf.estimator and predict on input data
  5. Keras : How should I prepare input data for RNN?
  6. What’s the difference of name scope and a variable scope in tensorflow?
  7. How does TensorFlow name tensors?
  8. What is the meaning of the “None” in model.summary of KERAS?
  9. Module ‘tensorflow’ has no attribute ‘contrib’
  10. Understanding TensorBoard (weight) histograms
  11. Confused by the behavior of `tf.cond`
  12. Multivariate LSTM with missing values
  13. TensorFlow: numpy.repeat() alternative
  14. TensorFlow, why there are 3 files after saving the model?
  15. Holding variables constant during optimizer
  16. Create keras callback to save model predictions and targets for each batch during training
  17. How to extract data/labels back from TensorFlow dataset
  18. Obtaining output of an Intermediate layer in TensorFlow/Keras
  19. Making predictions with a TensorFlow model
  20. tf.data with multiple inputs / outputs in Keras
  21. ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory
  22. Could not install packages due to an EnvironmentError: [WinError 5] Access is denied:
  23. CUDA_ERROR_OUT_OF_MEMORY in tensorflow
  24. Convert between NHWC and NCHW in TensorFlow
  25. How to properly use tf.metrics.accuracy?
  26. How can I convert a trained Tensorflow model to Keras?
  27. ImportError: libcudnn when running a TensorFlow program
  28. Google Storage (gs) wrapper file input/out for Cloud ML?
  29. parallelising tf.data.Dataset.from_generator
  30. Choosing number of Steps per Epoch

Leave a Comment