ResNet: 100% accuracy during training, but 33% prediction accuracy with the same data

by

It’s because of the batch normalization layers.

In training phase, the batch is normalized w.r.t. its mean and variance. However, in testing phase, the batch is normalized w.r.t. the moving average of previously observed mean and variance.

Now this is a problem when the number of observed batches is small (e.g., 5 in your example) because in the BatchNormalization layer, by default moving_mean is initialized to be 0 and moving_variance is initialized to be 1.

Given also that the default momentum is 0.99, you’ll need to update the moving averages quite a lot of times before they converge to the “real” mean and variance.

That’s why the prediction is wrong in the early stage, but is correct after 1000 epochs.

You can verify it by forcing the BatchNormalization layers to operate in “training mode”.

During training, the accuracy is 1 and the loss is close to zero:

model.fit(imgs,y,epochs=5,shuffle=True)
Epoch 1/5
3/3 [==============================] - 19s 6s/step - loss: 1.4624 - acc: 0.3333
Epoch 2/5
3/3 [==============================] - 0s 63ms/step - loss: 0.6051 - acc: 0.6667
Epoch 3/5
3/3 [==============================] - 0s 57ms/step - loss: 0.2168 - acc: 1.0000
Epoch 4/5
3/3 [==============================] - 0s 56ms/step - loss: 1.1921e-07 - acc: 1.0000
Epoch 5/5
3/3 [==============================] - 0s 53ms/step - loss: 1.1921e-07 - acc: 1.0000

Now if we evaluate the model, we’ll observe high loss and low accuracy because after 5 updates, the moving averages are still pretty close to the initial values:

model.evaluate(imgs,y)
3/3 [==============================] - 3s 890ms/step
[10.745396614074707, 0.3333333432674408]

However, if we manually specify the “learning phase” variable and let the BatchNormalization layers use the “real” batch mean and variance, the result becomes the same as what’s observed in fit().

sample_weights = np.ones(3)
learning_phase = 1  # 1 means "training"
ins = [imgs, y, sample_weights, learning_phase]
model.test_function(ins)
[1.192093e-07, 1.0]

It’s also possible to verify it by changing the momentum to a smaller value.

For example, by adding momentum=0.01 to all the batch norm layers in ResNet50, the prediction after 20 epochs is:

model.predict(imgs)
array([[  1.00000000e+00,   1.34882026e-08,   3.92139575e-22],
       [  0.00000000e+00,   1.00000000e+00,   0.00000000e+00],
       [  8.70998792e-06,   5.31159838e-10,   9.99991298e-01]], dtype=float32)

More Related Contents:

  1. Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?
  2. Many to one and many to many LSTM examples in Keras
  3. How to calculate optimal batch size
  4. What is the difference between loss function and metric in Keras? [duplicate]
  5. Keras Text Preprocessing – Saving Tokenizer object to file for scoring
  6. Is deep learning bad at fitting simple non linear functions outside training scope (extrapolating)?
  7. Can Keras deal with input images with different size?
  8. Keras: model.evaluate vs model.predict accuracy difference in multi-class NLP task
  9. What is the difference between Keras model.evaluate() and model.predict()?
  10. Which algorithms involves in deep learning? [closed]
  11. Common causes of nans during training of neural networks
  12. How to interpret MSE in Keras Regressor
  13. Intuitive understanding of 1D, 2D, and 3D convolutions in convolutional neural networks [closed]
  14. Caffe | solver.prototxt values setting strategy
  15. Epoch vs Iteration when training neural networks [closed]
  16. Does Any one got “AttributeError: ‘str’ object has no attribute ‘decode’ ” , while Loading a Keras Saved Model
  17. What is `weight_decay` meta parameter in Caffe?
  18. Keras custom loss function: Accessing current input pattern
  19. Calculate the output size in convolution layer [closed]
  20. Removing then Inserting a New Middle Layer in a Keras Model
  21. multi-layer perceptron (MLP) architecture: criteria for choosing number of hidden layers and size of the hidden layer? [closed]
  22. What is the role of TimeDistributed layer in Keras?
  23. TimeDistributed(Dense) vs Dense in Keras – Same number of parameters
  24. Fine Tuning of GoogLeNet Model
  25. Caffe: What can I do if only a small batch fits into memory?
  26. Keras, Tensorflow : Merge two different model output into one
  27. How to reduce a fully-connected (`”InnerProduct”`) layer using truncated SVD
  28. What is `lr_policy` in Caffe?
  29. Early stopping with Keras and sklearn GridSearchCV cross-validation
  30. How to fix “ResourceExhaustedError: OOM when allocating tensor”

Leave a Comment