Pandas Correlation Groupby

by

You pretty much figured out all the pieces, just need to combine them:

>>> df.groupby('ID')[['Val1','Val2']].corr()

             Val1      Val2
ID                         
A  Val1  1.000000  0.500000
   Val2  0.500000  1.000000
B  Val1  1.000000  0.385727
   Val2  0.385727  1.000000

In your case, printing out a 2×2 for each ID is excessively verbose. I don’t see an option to print a scalar correlation instead of the whole matrix, but you can do something simple like this if you only have two variables:

>>> df.groupby('ID')[['Val1','Val2']].corr().iloc[0::2,-1]

ID       
A   Val1    0.500000
B   Val1    0.385727

For the more general case of 3+ variables

For 3 or more variables, it is not straightforward to create concise output but you could do something like this:

groups = list('Val1', 'Val2', 'Val3', 'Val4')
df2 = pd.DataFrame()
for i in range( len(groups)-1): 
    df2 = df2.append( df.groupby('ID')[groups].corr().stack()
                        .loc[:,groups[i],groups[i+1]:].reset_index() )

df2.columns = ['ID', 'v1', 'v2', 'corr']
df2.set_index(['ID','v1','v2']).sort_index()

Note that if we didn’t have the groupby element, it would be straightforward to use an upper or lower triangle function from numpy. But since that element is present, it is not so easy to produce concise output in a more elegant manner as far as I can tell.

More Related Contents:

  1. Get statistics for each group (such as count, mean, etc) using pandas GroupBy?
  2. How do I create a new column from the output of pandas groupby().sum()?
  3. Apply multiple functions to multiple groupby columns
  4. Python Pandas : group by in group by and average?
  5. Pandas GroupBy.apply method duplicates first group
  6. Pandas dataframe get first row of each group
  7. Pandas groupby multiple fields then diff
  8. Pandas Groupby Range of Values
  9. How to access pandas groupby dataframe by key
  10. group by pandas dataframe and select latest in each group
  11. Pandas GroupBy and select rows with the minimum value in a specific column
  12. How to group a pandas dataframe by a defined time interval?
  13. Pandas: resample timeseries with groupby
  14. Make Pandas groupby act similarly to itertools groupby
  15. Python pandas – filter rows after groupby
  16. Pandas groupby nlargest sum
  17. Python Pandas How to assign groupby operation results back to columns in parent dataframe?
  18. why does pandas rolling use single dimension ndarray
  19. How to calculate correlation between all columns and remove highly correlated ones using pandas?
  20. pandas – find first occurrence
  21. python pandas groupby() result
  22. Add a sequential counter column on groups to a pandas dataframe
  23. Group dataframe and get sum AND count?
  24. Pivot Tables or Group By for Pandas?
  25. Add a sequence number to each element in a group using python
  26. Pandas groupby and aggregation output should include all the original columns (including the ones not aggregated on)
  27. Get the row corresponding to the max in pandas GroupBy [duplicate]
  28. Get frequency of item occurrences in a column as percentage [duplicate]
  29. Pandas: groupby forward fill with datetime index
  30. Why do I get a KeyError when using pandas apply?

Leave a Comment