Strange behaviour of the loss function in keras model, with pretrained convolutional base

by

Looks like I found the solution. As I have suggested the problem is with BatchNormalization layers. They make tree things

  1. subtract mean and normalize by std
  2. collect statistics on mean and std using running average
  3. train two additional parameters (two per node).

When one sets trainable to False, these two parameters freeze and layer also stops collecting statistic on mean and std. But it looks like the layer still performs normalization during training time using the training batch. Most likely it’s a bug in keras or maybe they did it on purpose for some reason. As a result the calculations on forward propagation during training time are different as compared with prediction time even though the trainable atribute is set to False.

There are two possible solutions i can think of:

  1. To set all BatchNormalization layers to trainable. In this case these layers will collect statistics from your dataset instead of using pretrained one (which can be significantly different!). In this case you will adjust all the BatchNorm layers to your custom dataset during the training.
  2. Split the model in two parts model=model_base+model_top. After that, use model_base to extract features by model_base.predict() and then feed these features into model_top and train only the model_top.

I’ve just tried the first solution and it looks like it’s working:

model.fit(x=dat[0],y=dat[1],batch_size=32)

Epoch 1/1
32/32 [==============================] - 1s 28ms/step - loss: **3.1053**

model.evaluate(x=dat[0],y=dat[1])

32/32 [==============================] - 0s 10ms/step
**2.487905502319336**

This was after some training – one need to wait till enough statistics on mean and std are collected.

Second solution i haven’t tried yet, but i’m pretty sure it’s gonna work since forward propagation during training and prediction will be the same.

Update. I found a great blog post where this issue has been discussed in all the details. Check it out here

More Related Contents:

  1. Keras, How to get the output of each layer?
  2. How to apply gradient clipping in TensorFlow?
  3. Keras replacing input layer
  4. Keras custom loss function: Accessing current input pattern
  5. How can I use a pre-trained neural network with grayscale images?
  6. Dimension of shape in conv1D
  7. Removing then Inserting a New Middle Layer in a Keras Model
  8. Custom loss function in Keras
  9. How do I get the weights of a layer in Keras?
  10. Multiple outputs in Keras
  11. ValueError: Input 0 of layer sequential is incompatible with the layer: : expected min_ndim=4, found ndim=3. Full shape received: [8, 28, 28]
  12. Keras LSTM input dimension setting
  13. How to implement Grad-CAM on a trained network
  14. How to fix “ResourceExhaustedError: OOM when allocating tensor”
  15. How do I plot a Keras/Tensorflow subclassing API model?
  16. Multi-input Multi-output Model with Keras Functional API
  17. Understanding Keras LSTMs
  18. Make a custom loss function in keras
  19. Keras not training on entire dataset
  20. Could not load dynamic library ‘cudart64_101.dll’ on tensorflow CPU-only installation
  21. Keras Sequential model with multiple inputs
  22. expected ndim=3, found ndim=2
  23. How to get other metrics in Tensorflow 2.0 (not only accuracy)?
  24. TensorFlow ValueError: Cannot feed value of shape (64, 64, 3) for Tensor u’Placeholder:0′, which has shape ‘(?, 64, 64, 3)’
  25. Unbalanced data and weighted cross entropy
  26. Save and load model optimizer state
  27. Keras misinterprets training data shape
  28. NotImplementedError: Cannot convert a symbolic Tensor (2nd_target:0) to a numpy array
  29. looking for source code of from gen_nn_ops in tensorflow
  30. How to use TensorFlow metrics in Keras

Leave a Comment