Detect and exclude outliers in a pandas DataFrame

by

If you have multiple columns in your dataframe and would like to remove all rows that have outliers in at least one column, the following expression would do that in one shot.

df = pd.DataFrame(np.random.randn(100, 3))

from scipy import stats
df[(np.abs(stats.zscore(df)) < 3).all(axis=1)]

description:

  • For each column, it first computes the Z-score of each value in the
    column, relative to the column mean and standard deviation.
  • It then takes the absolute Z-score because the direction does not
    matter, only if it is below the threshold.
  • all(axis=1) ensures that for each row, all column satisfy the
    constraint.
  • Finally, the result of this condition is used to index the dataframe.

Filter other columns based on a single column

  • Specify a column for the zscore, df[0] for example, and remove .all(axis=1).
df[(np.abs(stats.zscore(df[0])) < 3)]

More Related Contents:

  1. Truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()
  2. Logical operators for Boolean indexing in Pandas
  3. Filtering Pandas DataFrames on dates
  4. How do I filter a pandas DataFrame based on value counts?
  5. How to drop rows of Pandas DataFrame whose value in a certain column is NaN
  6. Annotate bars with values on Pandas bar plots
  7. Compare two columns using pandas
  8. Pandas column access w/column names containing spaces
  9. Add x and y labels to a pandas plot
  10. python pandas extract year from datetime: df[‘year’] = df[‘date’].year is not working
  11. Pandas DataFrame stack multiple column values into single column
  12. How to replace negative numbers in Pandas Data Frame by zero
  13. Removing the timestamp from a datetime in pandas dataframe
  14. Convert pandas data frame to series
  15. vectorize conditional assignment in pandas dataframe
  16. Pandas DataFrame merge summing column
  17. Binning a column with pandas
  18. Lambda including if…elif…else
  19. How can I replicate rows of a Pandas DataFrame?
  20. How to get/set a pandas index column title or name?
  21. Pandas DataFrame sort by categorical column but by specific class ordering
  22. Python pandas integer YYYYMMDD to datetime
  23. Pandas: drop columns with all NaN’s
  24. In Python pandas, start row index from 1 instead of zero without creating additional column
  25. Remove ends of string entries in pandas DataFrame column
  26. Collapsing rows in a Pandas dataframe if all rows have only one value in their columns
  27. start index at 1 for Pandas DataFrame
  28. Multi Index Sorting in Pandas
  29. Pythonic/efficient way to strip whitespace from every Pandas Data frame cell that has a stringlike object in it
  30. Difference between df[x], df[[x]], df[‘x’] , df[[‘x’]] and df.x

Leave a Comment