Epoch vs Iteration when training neural networks [closed]

by

In the neural network terminology:

  • one epoch = one forward pass and one backward pass of all the training examples
  • batch size = the number of training examples in one forward/backward pass. The higher the batch size, the more memory space you’ll need.
  • number of iterations = number of passes, each pass using [batch size] number of examples. To be clear, one pass = one forward pass + one backward pass (we do not count the forward pass and backward pass as two different passes).

For example: if you have 1000 training examples, and your batch size is 500, then it will take 2 iterations to complete 1 epoch.

FYI: Tradeoff batch size vs. number of iterations to train a neural network

The term “batch” is ambiguous: some people use it to designate the entire training set, and some people use it to refer to the number of training examples in one forward/backward pass (as I did in this answer). To avoid that ambiguity and make clear that batch corresponds to the number of training examples in one forward/backward pass, one can use the term mini-batch.

More Related Contents:

  1. Is this possible to predict the lottery numbers (not the most accurate)? [closed]
  2. Artificial Neural Network Toplogy
  3. What is the role of the bias in neural networks? [closed]
  4. Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?
  5. Clarification on a Neural Net that plays Snake
  6. Common causes of nans during training of neural networks
  7. Many to one and many to many LSTM examples in Keras
  8. How to interpret caffe log with debug_info?
  9. Caffe | solver.prototxt values setting strategy
  10. What is `weight_decay` meta parameter in Caffe?
  11. How to calculate optimal batch size
  12. What is the difference between loss function and metric in Keras? [duplicate]
  13. Keras Text Preprocessing – Saving Tokenizer object to file for scoring
  14. Why should weights of Neural Networks be initialized to random numbers? [closed]
  15. Is deep learning bad at fitting simple non linear functions outside training scope (extrapolating)?
  16. multi-layer perceptron (MLP) architecture: criteria for choosing number of hidden layers and size of the hidden layer? [closed]
  17. How to interpret loss and accuracy for a machine learning model [closed]
  18. Caffe: What can I do if only a small batch fits into memory?
  19. How to create caffe.deploy from train.prototxt
  20. How to reduce a fully-connected (`”InnerProduct”`) layer using truncated SVD
  21. What is the difference between Keras model.evaluate() and model.predict()?
  22. What is `lr_policy` in Caffe?
  23. How to initialize weights in PyTorch?
  24. How do I initialize weights in PyTorch?
  25. Cost function training target versus accuracy desired goal
  26. How to create a new gym environment in OpenAI?
  27. Does Caffe need data to be shuffled?
  28. Can Keras deal with input images with different size?
  29. keras: how to save the training history attribute of the history object
  30. Choosing number of Steps per Epoch

Leave a Comment