difference between StratifiedKFold and StratifiedShuffleSplit in sklearn

by

In stratKFolds, each test set should not overlap, even when shuffle is included. With stratKFolds and shuffle=True, the data is shuffled once at the start, and then divided into the number of desired splits. The test data is always one of the splits, the train data is the rest.

In ShuffleSplit, the data is shuffled every time, and then split. This means the test sets may overlap between the splits.

See this block for an example of the difference. Note the overlap of the elements in the test sets for ShuffleSplit.

splits = 5

tx = range(10)
ty = [0] * 5 + [1] * 5

from sklearn.model_selection import StratifiedShuffleSplit, StratifiedKFold
from sklearn import datasets

stratKfold = StratifiedKFold(n_splits=splits, shuffle=True, random_state=42)
shufflesplit = StratifiedShuffleSplit(n_splits=splits, random_state=42, test_size=2)

print("stratKFold")
for train_index, test_index in stratKfold.split(tx, ty):
    print("TRAIN:", train_index, "TEST:", test_index)

print("Shuffle Split")
for train_index, test_index in shufflesplit.split(tx, ty):
    print("TRAIN:", train_index, "TEST:", test_index)

Output:

stratKFold
TRAIN: [0 2 3 4 5 6 7 9] TEST: [1 8]
TRAIN: [0 1 2 3 5 7 8 9] TEST: [4 6]
TRAIN: [0 1 3 4 5 6 8 9] TEST: [2 7]
TRAIN: [1 2 3 4 6 7 8 9] TEST: [0 5]
TRAIN: [0 1 2 4 5 6 7 8] TEST: [3 9]
Shuffle Split
TRAIN: [8 4 1 0 6 5 7 2] TEST: [3 9]
TRAIN: [7 0 3 9 4 5 1 6] TEST: [8 2]
TRAIN: [1 2 5 6 4 8 9 0] TEST: [3 7]
TRAIN: [4 6 7 8 3 5 1 2] TEST: [9 0]
TRAIN: [7 2 6 5 4 3 0 9] TEST: [1 8]

As for when to use them, I tend to use stratKFolds for any cross validation, and I use ShuffleSplit with a split of 2 for my train/test set splits. But I’m sure there are other use cases for both.

More Related Contents:

  1. Does GridSearchCV perform cross-validation?
  2. Model help using Scikit-learn when using GridSearch
  3. Save classifier to disk in scikit-learn
  4. Accuracy Score ValueError: Can’t Handle mix of binary and continuous target
  5. Is it possible to specify your own distance function using scikit-learn K-Means Clustering?
  6. How to get most informative features for scikit-learn classifiers?
  7. Why is Random Forest with a single tree much better than a Decision Tree classifier?
  8. How to get precision, recall and f-measure from confusion matrix in Python [duplicate]
  9. scikit-learn GridSearchCV with multiple repetitions
  10. Feature/Variable importance after a PCA analysis
  11. Cross-validation metrics in scikit-learn for each data split
  12. Recovering features names of explained_variance_ratio_ in PCA with sklearn
  13. Custom transformer for sklearn Pipeline that alters both X and y
  14. What is exactly sklearn.pipeline.Pipeline?
  15. Save MinMaxScaler model in sklearn
  16. confusion matrix error “Classification metrics can’t handle a mix of multilabel-indicator and multiclass targets”
  17. Scikit-learn: How to obtain True Positive, True Negative, False Positive and False Negative
  18. Using explicit (predefined) validation set for grid search with sklearn
  19. ValueError: n_splits=10 cannot be greater than the number of members in each class
  20. Can anyone explain me StandardScaler?
  21. What are the pros and cons between get_dummies (Pandas) and OneHotEncoder (Scikit-learn)?
  22. TfidfVectorizer in scikit-learn : ValueError: np.nan is an invalid document
  23. How to compute precision, recall, accuracy and f1-score for the multiclass case with scikit learn?
  24. scikit-learn: how to scale back the ‘y’ predicted result
  25. Using Smote with Gridsearchcv in Scikit-learn
  26. confused about random_state in decision tree of scikit learn
  27. Can I send callbacks to a KerasClassifier?
  28. How to get most informative features for scikit-learn classifier for different class?
  29. Does the SVM in sklearn support incremental (online) learning?
  30. scikit-learn .predict() default threshold

Leave a Comment