How to split/partition a dataset into training and test datasets for, e.g., cross validation?

by

If you want to split the data set once in two parts, you can use numpy.random.shuffle, or numpy.random.permutation if you need to keep track of the indices (remember to fix the random seed to make everything reproducible):

import numpy
# x is your dataset
x = numpy.random.rand(100, 5)
numpy.random.shuffle(x)
training, test = x[:80,:], x[80:,:]

or

import numpy
# x is your dataset
x = numpy.random.rand(100, 5)
indices = numpy.random.permutation(x.shape[0])
training_idx, test_idx = indices[:80], indices[80:]
training, test = x[training_idx,:], x[test_idx,:]

There are many ways other ways to repeatedly partition the same data set for cross validation. Many of those are available in the sklearn library (k-fold, leave-n-out, …). sklearn also includes more advanced “stratified sampling” methods that create a partition of the data that is balanced with respect to some features, for example to make sure that there is the same proportion of positive and negative examples in the training and test set.

More Related Contents:

  1. Concatenate range arrays given start, stop numbers in a vectorized way – NumPy
  2. Techniques for working with large Numpy arrays? [duplicate]
  3. Sorting arrays in NumPy by column
  4. How to access the ith column of a NumPy multidimensional array?
  5. Python/NumPy first occurrence of subarray
  6. Concatenating two one-dimensional NumPy arrays
  7. Numpy matrix to array
  8. Replacing Numpy elements if condition is met
  9. How to make a multidimension numpy array with a varying row size?
  10. Force NumPy ndarray to take ownership of its memory in Cython
  11. How to normalize a NumPy array to within a certain range?
  12. subsampling every nth entry in a numpy array
  13. Immutable numpy array?
  14. How to get a list of all indices of repeated elements in a numpy array
  15. Binary random array with a specific proportion of ones?
  16. Efficient way to take the minimum/maximum n values and indices from a matrix using NumPy
  17. Compress numpy arrays efficiently
  18. Add multiple values to one numpy array index
  19. Index 2D numpy array by a 2D array of indices without loops
  20. Flattening a list of NumPy arrays?
  21. Element-wise string concatenation in numpy
  22. Fastest pairwise distance metric in python
  23. Partition array into N chunks with Numpy
  24. numpy division with RuntimeWarning: invalid value encountered in double_scalars
  25. How does the axis parameter from NumPy work?
  26. numpy – evaluate function on a grid of points
  27. numpy arange: how to make “precise” array of floats?
  28. Interweaving two NumPy arrays efficiently
  29. Formatting floats in a numpy array [duplicate]
  30. How to convert a 3d numpy array to 2d

Leave a Comment