Common causes of nans during training of neural networks

I came across this phenomenon several times. Here are my observations: Gradient blow up Reason: large gradients throw the learning process off-track. What you should expect: Looking at the runtime log, you should look at the loss values per-iteration. You’ll notice that the loss starts to grow significantly from iteration to iteration, eventually the loss … Read more

A guide to convert_imageset.cpp

A quick guide to Caffe’s convert_imageset Build First thing you must do is build caffe and caffe’s tools (convert_imageset is one of these tools). After installing caffe and makeing it make sure you ran make tools as well. Verify that a binary file convert_imageset is created in $CAFFE_ROOT/build/tools. Prepare your data Images: put all images … Read more