What is right batch normalization function in Tensorflow?

by

Just to add to the list, there’re several more ways to do batch-norm in tensorflow:

  • tf.nn.batch_normalization is a low-level op. The caller is responsible to handle mean and variance tensors themselves.
  • tf.nn.fused_batch_norm is another low-level op, similar to the previous one. The difference is that it’s optimized for 4D input tensors, which is the usual case in convolutional neural networks. tf.nn.batch_normalization accepts tensors of any rank greater than 1.
  • tf.layers.batch_normalization is a high-level wrapper over the previous ops. The biggest difference is that it takes care of creating and managing the running mean and variance tensors, and calls a fast fused op when possible. Usually, this should be the default choice for you.
  • tf.contrib.layers.batch_norm is the early implementation of batch norm, before it’s graduated to the core API (i.e., tf.layers). The use of it is not recommended because it may be dropped in the future releases.
  • tf.nn.batch_norm_with_global_normalization is another deprecated op. Currently, delegates the call to tf.nn.batch_normalization, but likely to be dropped in the future.
  • Finally, there’s also Keras layer keras.layers.BatchNormalization, which in case of tensorflow backend invokes tf.nn.batch_normalization.

More Related Contents:

  1. How to make a custom activation function with only Python in Tensorflow?
  2. How to assign a value to a TensorFlow variable?
  3. Multiple outputs in Keras
  4. Keras, How to get the output of each layer?
  5. How to choose cross-entropy loss in TensorFlow?
  6. Using a pre-trained word embedding (word2vec or Glove) in TensorFlow
  7. What is the difference between ‘SAME’ and ‘VALID’ padding in tf.nn.max_pool of tensorflow?
  8. Keras + Tensorflow and Multiprocessing in Python
  9. Keras Sequential model input layer
  10. How to initialize weights in PyTorch?
  11. Tensorflow: How to replace or modify gradient?
  12. How do I initialize weights in PyTorch?
  13. Keras custom loss function: Accessing current input pattern
  14. Negative dimension size caused by subtracting 3 from 1 for ‘conv2d_2/convolution’
  15. How do I load a caffe model and convert to a numpy array?
  16. Dimension of shape in conv1D
  17. How to feed caffe multi label data in HDF5 format?
  18. Loading a trained Keras model and continue training
  19. How do I check if keras is using gpu version of tensorflow?
  20. How to set layer-wise learning rate in Tensorflow?
  21. What is the role of TimeDistributed layer in Keras?
  22. How to handle non-determinism when training on a GPU?
  23. ValueError: Input 0 of layer sequential is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [8, 28, 28]
  24. How to implement Grad-CAM on a trained network
  25. looking for source code of from gen_nn_ops in tensorflow
  26. ValueError: Tensor must be from the same graph as Tensor with Bidirectinal RNN in Tensorflow
  27. How to fix “ResourceExhaustedError: OOM when allocating tensor”
  28. How do I plot a Keras/Tensorflow subclassing API model?
  29. Strange behaviour of the loss function in keras model, with pretrained convolutional base
  30. Multi-input Multi-output Model with Keras Functional API

Leave a Comment