scale the loss value according to “badness” in caffe

by

You are correct in observing that the scaling you are doing for the log(prob) is exactly what "InfogainLoss" layer is doing (You can read more about it here and here).

As for the derivative (back-prop): the loss computed by this layer is 

L = - sum_j infogain_mat[label * dim + j] * log( prob(j) )

If you differentiate this expression with respect to prob(j) (which is the input variable to this layer), you’ll notice that the derivative of log(x) is 1/x this is why you see that

dL/dprob(j) = - infogain_mat[label * dim + j] / prob(j)

Now, why don’t you see similar expression in the back-prop of "SoftmaxWithLoss" layer?
well, as the name of that layer suggests it is actually a combination of two layers: softmax that computes class probabilities from classifiers outputs and a log loss layer on top of it. Combining these two layer enables a more numerically robust estimation of the gradients.
Working a little with "InfogainLoss" layer I noticed that sometimes prob(j) can have a very small value leading to unstable estimation of the gradients.

Here’s a detailed computation of the forward and backward passes of "SoftmaxWithLoss" and "InfogainLoss" layers with respect to the raw predictions (x), rather than the “softmax” probabilities derived from these predictions using a softmax layer. You can use these equations to create a "SoftmaxWithInfogainLoss" layer that is more numerically robust than computing infogain loss on top of a softmax layer:

enter image description here

PS,
Note that if you are going to use infogain loss for weighing, you should feed H (the infogain_mat) with label similarities, rather than distances.

Update:
I recently implemented this robust gradient computation and created this pull request. This PR was merged to master branch on April, 2017.

More Related Contents:

  1. Test labels for regression caffe, float not allowed?
  2. Caffe | solver.prototxt values setting strategy
  3. How to calculate the number of parameters for convolutional neural network?
  4. LSTM module for Caffe
  5. caffe data layer example step by step
  6. How to create caffe.deploy from train.prototxt
  7. caffe: model definition: write same layer with different phase using caffe.NetSpec()
  8. 2-D convolution as a matrix-matrix multiplication [closed]
  9. should I use float or classes as output for the final layer in my neural network?
  10. Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?
  11. Keras input explanation: input_shape, units, batch_size, dim, etc
  12. Common causes of nans during training of neural networks
  13. What is a `”Python”` layer in caffe?
  14. InfogainLoss layer
  15. How to interpret caffe log with debug_info?
  16. How to training/testing my own dataset in caffe?
  17. [caffe]: check fails: Check failed: hdf_blobs_[i]->shape(0) == num (200 vs. 6000)
  18. What is `weight_decay` meta parameter in Caffe?
  19. How do I load a caffe model and convert to a numpy array?
  20. How to feed caffe multi label data in HDF5 format?
  21. Does Caffe need data to be shuffled?
  22. Tackling Class Imbalance: scaling contribution to loss and sgd
  23. Fine Tuning of GoogLeNet Model
  24. Caffe: What can I do if only a small batch fits into memory?
  25. How to reduce a fully-connected (`”InnerProduct”`) layer using truncated SVD
  26. What is `lr_policy` in Caffe?
  27. Many to one and many to many LSTM examples in Keras
  28. What is the difference between loss function and metric in Keras? [duplicate]
  29. How to interpret loss and accuracy for a machine learning model [closed]
  30. What is right batch normalization function in Tensorflow?

Leave a Comment