Early stopping with Keras and sklearn GridSearchCV cross-validation

by

[Answer after the question was edited & clarified:]

Before rushing into implementation issues, it is always a good practice to take some time to think about the methodology and the task itself; arguably, intermingling early stopping with the cross validation procedure is not a good idea.

Let’s make up an example to highlight the argument.

Suppose that you indeed use early stopping with 100 epochs, and 5-fold cross validation (CV) for hyperparameter selection. Suppose also that you end up with a hyperparameter set X giving best performance, say 89.3% binary classification accuracy.

Now suppose that your second-best hyperparameter set, Y, gives 89.2% accuracy. Examining closely the individual CV folds, you see that, for your best case X, 3 out of the 5 CV folds exhausted the max 100 epochs, while in the other 2 early stopping kicked in, say in 95 and 93 epochs respectively.

Now imagine that, examining your second-best set Y, you see that again 3 out of the 5 CV folds exhausted the 100 epochs, while the other 2 both stopped early enough at ~ 80 epochs.

What would be your conclusion from such an experiment?

Arguably, you would have found yourself in an inconclusive situation; further experiments might reveal which is actually the best hyperparameter set, provided of course that you would have thought to look into these details of the results in the first place. And needless to say, if all this was automated through a callback, you might have missed your best model despite the fact that you would have actually tried it.

The whole CV idea is implicitly based on the “all other being equal” argument (which of course is never true in practice, only approximated in the best possible way). If you feel that the number of epochs should be a hyperparameter, just include it explicitly in your CV as such, rather than inserting it through the back door of early stopping, thus possibly compromising the whole process (not to mention that early stopping has itself a hyperparameter, patience).

Not intermingling these two techniques doesn’t mean of course that you cannot use them sequentially: once you have obtained your best hyperparameters through CV, you can always employ early stopping when fitting the model in your whole training set (provided of course that you do have a separate validation set).

The field of deep neural nets is still (very) young, and it is true that it has yet to establish its “best practice” guidelines; add the fact that, thanks to an amazing community, there are all sort of tools available in open source implementations, and you can easily find yourself into the (admittedly tempting) position of mixing things up just because they happen to be available. I am not necessarily saying that this is what you are attempting to do here – I am just urging for more caution when combining ideas that may have not been designed to work along together…

More Related Contents:

  1. How to interpret MSE in Keras Regressor
  2. Does GridSearchCV perform cross-validation?
  3. Model help using Scikit-learn when using GridSearch
  4. Is this possible to predict the lottery numbers (not the most accurate)? [closed]
  5. What function defines accuracy in Keras when the loss is mean squared error (MSE)?
  6. Why binary_crossentropy and categorical_crossentropy give different performances for the same problem?
  7. Loss & accuracy – Are these reasonable learning curves?
  8. Many to one and many to many LSTM examples in Keras
  9. scikit-learn GridSearchCV with multiple repetitions
  10. Getting a low ROC AUC score but a high accuracy
  11. Order between using validation, training and test sets
  12. How to calculate optimal batch size
  13. What is the difference between loss function and metric in Keras? [duplicate]
  14. Keras Text Preprocessing – Saving Tokenizer object to file for scoring
  15. difference between StratifiedKFold and StratifiedShuffleSplit in sklearn
  16. ResNet: 100% accuracy during training, but 33% prediction accuracy with the same data
  17. Is deep learning bad at fitting simple non linear functions outside training scope (extrapolating)?
  18. confusion matrix error “Classification metrics can’t handle a mix of multilabel-indicator and multiclass targets”
  19. How to put more weight on certain features in machine learning?
  20. Can Keras deal with input images with different size?
  21. TimeDistributed(Dense) vs Dense in Keras – Same number of parameters
  22. Keras: model.evaluate vs model.predict accuracy difference in multi-class NLP task
  23. What is the difference between Keras model.evaluate() and model.predict()?
  24. Using Smote with Gridsearchcv in Scikit-learn
  25. Scikits-Learn RandomForrest trained on 64bit python wont open on 32bit python
  26. Can I send callbacks to a KerasClassifier?
  27. Scikit learn – fit_transform on the test set
  28. How can I get biases from a trained model in Keras?
  29. How should I teach machine learning algorithm using data with big disproportion of classes? (SVM)
  30. Under what parameters are SVC and LinearSVC in scikit-learn equivalent?

Leave a Comment