Is there a numpy builtin to reject outliers from a list

by

Something important when dealing with outliers is that one should try to use estimators as robust as possible. The mean of a distribution will be biased by outliers but e.g. the median will be much less.

Building on eumiro’s answer:

def reject_outliers(data, m = 2.):
    d = np.abs(data - np.median(data))
    mdev = np.median(d)
    s = d/mdev if mdev else 0.
    return data[s<m]

Here I have replace the mean with the more robust median and the standard deviation with the median absolute distance to the median. I then scaled the distances by their (again) median value so that m is on a reasonable relative scale.

Note that for the data[s<m] syntax to work, data must be a numpy array.

More Related Contents:

  1. Is there a NumPy function to return the first index of something in an array?
  2. Understanding NumPy’s einsum
  3. How to calculate rolling / moving average using python + NumPy / SciPy?
  4. Saving a Numpy array as an image
  5. Removing nan values from an array
  6. How to count the occurrence of certain item in an ndarray?
  7. How do I compute derivative using Numpy?
  8. Numpy: Get random set of rows from 2D array
  9. How can I release memory after creating matplotlib figures
  10. Sliding window of M-by-N shape numpy.ndarray
  11. Precision lost while using read_csv in pandas
  12. Improving FFT performance in Python
  13. reading v 7.3 mat file in python
  14. Creating a Confidence Ellipses in a sccatterplot using matplotlib
  15. Numpy: find index of the elements within range
  16. Create new column based on values from other columns / apply a function of multiple columns, row-wise in Pandas
  17. Pandas groupby with categories with redundant nan
  18. Easy way to test if each element in an numpy array lies between two values?
  19. Python numpy: cannot convert datetime64[ns] to datetime64[D] (to use with Numba)
  20. ImportError: No module named numpy on spark workers
  21. Automatic White Balancing with Grayworld assumption
  22. C++ conversion from NumPy array to Mat (OpenCV)
  23. Weird behaviour initializing a numpy array of string data
  24. Extract csv file specific columns to list in Python
  25. How to “scale” a numpy array?
  26. How to convert an image from np.uint16 to np.uint8?
  27. Python import error: cannot import name ‘six’ from ‘sklearn.externals’
  28. Passing C++ vector to Numpy through Cython without copying and taking care of memory management automatically
  29. Numpy Broadcast to perform euclidean distance vectorized
  30. ValueError: x and y must be the same size

Leave a Comment