How to calculate correlation between all columns and remove highly correlated ones using pandas?

by

The method here worked well for me, only a few lines of code: https://chrisalbon.com/machine_learning/feature_selection/drop_highly_correlated_features/

import numpy as np

# Create correlation matrix
corr_matrix = df.corr().abs()

# Select upper triangle of correlation matrix
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))

# Find features with correlation greater than 0.95
to_drop = [column for column in upper.columns if any(upper[column] > 0.95)]

# Drop features 
df.drop(to_drop, axis=1, inplace=True)

More Related Contents:

  1. Use .corr to get the correlation between two columns
  2. Pandas Correlation Groupby
  3. pandas columns correlation with statistical significance
  4. how to calculation cost time [closed]
  5. GroupBy pandas DataFrame and select most common value
  6. String concatenation of two pandas columns
  7. pandas: filter rows of DataFrame with operator chaining
  8. Reshape wide to long in pandas
  9. How do I create test and train samples from one dataframe with pandas?
  10. pandas groupby sort within groups
  11. Applying function with multiple arguments to create a new pandas column
  12. How to create a large pandas dataframe from an sql query without running out of memory?
  13. How do convert a pandas dataframe to XML?
  14. How to add hovering annotations to a plot
  15. Sort a pandas dataframe series by month name
  16. Count frequency of values in pandas DataFrame column
  17. append dataframe to excel with pandas
  18. Strip / trim all strings of a dataframe
  19. How to dynamically update a plot in a loop in IPython notebook (within one cell)
  20. How to iterate over consecutive chunks of Pandas dataframe efficiently
  21. How to flatten multilevel/nested JSON?
  22. Neural Network LSTM input shape from dataframe
  23. Joining pandas DataFrames by Column names
  24. Get list of pandas dataframe columns based on data type
  25. Pandas Dataframe: split column into multiple columns, right-align inconsistent cell entries
  26. Pandas in AWS lambda gives numpy error
  27. Pandas: Reading Excel with merged cells
  28. Pandas groupby with categories with redundant nan
  29. Writing a Python Pandas DataFrame to Word document
  30. How to annotate a seaborn barplot with the aggregated value

Leave a Comment