scikit-learn GridSearchCV with multiple repetitions

by

This is called as nested cross_validation. You can look at official documentation example to guide you into right direction and also have a look at my other answer here for a similar approach.

You can adapt the steps to suit your need:

svr = SVC(kernel="rbf")
c_grid = {"C": [1, 10, 100, ...  ]}

# CV Technique "LabelKFold", "LeaveOneOut", "LeaveOneLabelOut", etc.

# To be used within GridSearch (5 in your case)
inner_cv = KFold(n_splits=5, shuffle=True, random_state=i)

# To be used in outer CV (you asked for 10)
outer_cv = KFold(n_splits=10, shuffle=True, random_state=i)

# Non_nested parameter search and scoring
clf = GridSearchCV(estimator=svr, param_grid=c_grid, cv=inner_cv)
clf.fit(X_iris, y_iris)
non_nested_score = clf.best_score_

# Pass the gridSearch estimator to cross_val_score
# This will be your required 10 x 5 cvs
# 10 for outer cv and 5 for gridSearch's internal CV
clf = GridSearchCV(estimator=svr, param_grid=c_grid, cv=inner_cv)
nested_score = cross_val_score(clf, X=X_iris, y=y_iris, cv=outer_cv).mean()

Edit – Description of nested cross validation with cross_val_score() and GridSearchCV()

  1. clf = GridSearchCV(estimator, param_grid, cv= inner_cv).
  2. Pass clf, X, y, outer_cv to cross_val_score
  3. As seen in source code of cross_val_score, this X will be divided into X_outer_train, X_outer_test using outer_cv. Same for y.
  4. X_outer_test will be held back and X_outer_train will be passed on to clf for fit() (GridSearchCV in our case). Assume X_outer_train is called X_inner from here on since it is passed to inner estimator, assume y_outer_train is y_inner.
  5. X_inner will now be split into X_inner_train and X_inner_test using inner_cv in the GridSearchCV. Same for y
  6. Now the gridSearch estimator will be trained using X_inner_train and y_train_inner and scored using X_inner_test and y_inner_test.
  7. The steps 5 and 6 will be repeated for inner_cv_iters (5 in this case).
  8. The hyper-parameters for which the average score over all inner iterations (X_inner_train, X_inner_test) is best, is passed on to the clf.best_estimator_ and fitted for all data, i.e. X_outer_train.
  9. This clf (gridsearch.best_estimator_) will then be scored using X_outer_test and y_outer_test.
  10. The steps 3 to 9 will be repeated for outer_cv_iters (10 here) and array of scores will returned from cross_val_score
  11. We then use mean() to get back nested_score.

More Related Contents:

  1. Does GridSearchCV perform cross-validation?
  2. Model help using Scikit-learn when using GridSearch
  3. Cross-validation metrics in scikit-learn for each data split
  4. scikit-learn cross validation, negative values with mean squared error
  5. difference between StratifiedKFold and StratifiedShuffleSplit in sklearn
  6. Using explicit (predefined) validation set for grid search with sklearn
  7. ValueError: n_splits=10 cannot be greater than the number of members in each class
  8. Using Smote with Gridsearchcv in Scikit-learn
  9. sklearn error ValueError: Input contains NaN, infinity or a value too large for dtype(‘float64’)
  10. Save classifier to disk in scikit-learn
  11. where to put freeze_support() in a Python script?
  12. How to do n-D distance and nearest neighbor calculations on numpy arrays
  13. Is there a library function for Root mean square error (RMSE) in python?
  14. How to get precision, recall and f-measure from confusion matrix in Python [duplicate]
  15. How to convert a Scikit-learn dataset to a Pandas dataset
  16. sklearn.LabelEncoder with never seen before values
  17. Custom transformer for sklearn Pipeline that alters both X and y
  18. Scikit Learn SVC decision_function and predict
  19. RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  20. Save MinMaxScaler model in sklearn
  21. sklearn plot confusion matrix with labels
  22. confusion matrix error “Classification metrics can’t handle a mix of multilabel-indicator and multiclass targets”
  23. What are the pros and cons between get_dummies (Pandas) and OneHotEncoder (Scikit-learn)?
  24. Run an OLS regression with Pandas Data Frame
  25. How to upgrade scikit-learn package in anaconda
  26. sklearn pipeline – how to apply different transformations on different columns
  27. Mixing categorial and continuous data in Naive Bayes classifier using scikit-learn
  28. Early stopping with Keras and sklearn GridSearchCV cross-validation
  29. Using scikit to determine contributions of each feature to a specific class prediction
  30. Determining the most contributing features for SVM classifier in sklearn

Leave a Comment