Order between using validation, training and test sets

by

The Wikipedia article is not wrong; according to my own experience, this is a frequent point of confusion among newcomers to ML.

There are two separate ways of approaching the problem:

  • Either you use an explicit validation set to do hyperparameter search & tuning
  • Or you use cross-validation

So, the standard point is that you always put aside a portion of your data as test set; this is used for no other reason than assessing the performance of your model in the end (i.e. not back-and-forth and multiple assessments, because in that case you are using your test set as a validation set, which is bad practice).

After you have done that, you choose if you will cut another portion of your remaining data to use as a separate validation set, or if you will proceed with cross-validation (in which case, no separate and fixed validation set is required).

So, essentially, both your first and third approaches are valid (and mutually exclusive, i.e. you should choose which one you will go with). The second one, as you describe it (CV only in the validation set?), is certainly not (as said, when you choose to go with CV you don’t assign a separate validation set). Apart from a brief mention of cross-validation, what the Wikipedia article actually describes is your first approach.

Questions of which approach is “better” cannot of course be answered at that level of generality; both approaches are indeed valid, and are used depending on the circumstances. Very loosely speaking, I would say that in most “traditional” (i.e. non deep learning) ML settings, most people choose to go with cross-validation; but there are cases where this is not practical (most deep learning settings, again loosely speaking), and people are going with a separate validation set instead.

More Related Contents:

  1. Should Feature Selection be done before Train-Test Split or after?
  2. Early stopping with Keras and sklearn GridSearchCV cross-validation
  3. Is this possible to predict the lottery numbers (not the most accurate)? [closed]
  4. Machine understand natural language – nlp [closed]
  5. Speeker Identification (Machine Learning)
  6. Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?
  7. Is there a rule-of-thumb for how to divide a dataset into training and validation sets? [closed]
  8. What is the difference between steps and epochs in TensorFlow?
  9. Getting a low ROC AUC score but a high accuracy
  10. What is the difference between a sigmoid followed by the cross entropy and sigmoid_cross_entropy_with_logits in TensorFlow?
  11. What is `weight_decay` meta parameter in Caffe?
  12. Why does one hot encoding improve machine learning performance? [closed]
  13. How to calculate optimal batch size
  14. Train Tensorflow Object Detection on own dataset
  15. Keras Text Preprocessing – Saving Tokenizer object to file for scoring
  16. Unable to approximate the sine function using a neural network
  17. difference between StratifiedKFold and StratifiedShuffleSplit in sklearn
  18. Calculate the output size in convolution layer [closed]
  19. Why should weights of Neural Networks be initialized to random numbers? [closed]
  20. multi-layer perceptron (MLP) architecture: criteria for choosing number of hidden layers and size of the hidden layer? [closed]
  21. Pattern recognition in time series [closed]
  22. Fine Tuning of GoogLeNet Model
  23. Caffe: What can I do if only a small batch fits into memory?
  24. How to reduce a fully-connected (`”InnerProduct”`) layer using truncated SVD
  25. What is the difference between supervised learning and unsupervised learning? [closed]
  26. Which machine learning classifier to choose, in general? [closed]
  27. Using pre-trained word2vec with LSTM for word generation
  28. How can I get biases from a trained model in Keras?
  29. Model help using Scikit-learn when using GridSearch
  30. Under what parameters are SVC and LinearSVC in scikit-learn equivalent?

Leave a Comment