What is `weight_decay` meta parameter in Caffe?

by

The weight_decay meta parameter govern the regularization term of the neural net.

During training a regularization term is added to the network’s loss to compute the backprop gradient. The weight_decay value determines how dominant this regularization term will be in the gradient computation.

As a rule of thumb, the more training examples you have, the weaker this term should be. The more parameters you have (i.e., deeper net, larger filters, larger InnerProduct layers etc.) the higher this term should be.

Caffe also allows you to choose between L2 regularization (default) and L1 regularization, by setting

regularization_type: "L1"

However, since in most cases weights are small numbers (i.e., -1<w<1), the L2 norm of the weights is significantly smaller than their L1 norm. Thus, if you choose to use regularization_type: "L1" you might need to tune weight_decay to a significantly smaller value.

While learning rate may (and usually does) change during training, the regularization weight is fixed throughout.

More Related Contents:

  1. Common causes of nans during training of neural networks
  2. How to interpret caffe log with debug_info?
  3. Caffe: What can I do if only a small batch fits into memory?
  4. What is `lr_policy` in Caffe?
  5. Caffe | solver.prototxt values setting strategy
  6. How to calculate optimal batch size
  7. How to create caffe.deploy from train.prototxt
  8. How to reduce a fully-connected (`”InnerProduct”`) layer using truncated SVD
  9. Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?
  10. What is a `”Python”` layer in caffe?
  11. Many to one and many to many LSTM examples in Keras
  12. Epoch vs Iteration when training neural networks [closed]
  13. Cost function training target versus accuracy desired goal
  14. What is the difference between loss function and metric in Keras? [duplicate]
  15. Keras Text Preprocessing – Saving Tokenizer object to file for scoring
  16. Does Caffe need data to be shuffled?
  17. Why should weights of Neural Networks be initialized to random numbers? [closed]
  18. Tackling Class Imbalance: scaling contribution to loss and sgd
  19. Is deep learning bad at fitting simple non linear functions outside training scope (extrapolating)?
  20. multi-layer perceptron (MLP) architecture: criteria for choosing number of hidden layers and size of the hidden layer? [closed]
  21. How to interpret loss and accuracy for a machine learning model [closed]
  22. Fine Tuning of GoogLeNet Model
  23. What is the difference between Keras model.evaluate() and model.predict()?
  24. What is the role of the bias in neural networks? [closed]
  25. Loss & accuracy – Are these reasonable learning curves?
  26. How to initialize weights in PyTorch?
  27. How do I initialize weights in PyTorch?
  28. Why do we need to call zero_grad() in PyTorch?
  29. Can Keras deal with input images with different size?
  30. Keras: model.evaluate vs model.predict accuracy difference in multi-class NLP task

Leave a Comment