Faster way to transform group with mean value in Pandas

by

Current method, using transform

In [44]: grp = df["signal"].groupby(g)

In [45]: result2 = df["signal"].groupby(g).transform(np.mean)

In [47]: %timeit df["signal"].groupby(g).transform(np.mean)
1 loops, best of 3: 535 ms per loop

Using ‘broadcasting’ of the results

 In [43]: result = pd.concat([ Series([r]*len(grp.groups[i])) for i, r in enumerate(grp.mean().values) ],ignore_index=True)

In [42]: %timeit pd.concat([ Series([r]*len(grp.groups[i])) for i, r in enumerate(grp.mean().values) ],ignore_index=True)
10 loops, best of 3: 119 ms per loop

In [46]: result.equals(result2)
Out[46]: True

I think you might need to set the index of the returned on the broadcast result (it happens to work here because its a default index

result = pd.concat([ Series([r]*len(grp.groups[i])) for i, r in enumerate(grp.mean().values) ],ignore_index=True)
result.index = df.index

More Related Contents:

  1. Replacing values represented by 'UN' with NaN
  2. Convert pandas dataframe to NumPy array
  3. FutureWarning: elementwise comparison failed; returning scalar, but in the future will perform elementwise comparison
  4. pandas select from Dataframe using startswith
  5. What’s the fastest way in Python to calculate cosine similarity given sparse matrix data?
  6. Restart cumsum and get index if cumsum more than value
  7. How can I map True/False to 1/0 in a Pandas DataFrame?
  8. What does the term “broadcasting” mean in Pandas documentation?
  9. Dropping infinite values from dataframes in pandas?
  10. ValueError: numpy.dtype has the wrong size, try recompiling
  11. Pandas: convert dtype ‘object’ to int
  12. Pandas lookup from one of multiple columns, based on value
  13. Most efficient way to sort an array into bins specified by an index array?
  14. Pandas pd.Series.isin performance with set versus array
  15. Difference between data type ‘datetime64[ns]’ and ‘
  16. python pandas flatten a dataframe to a list
  17. Get week start date (Monday) from a date column in Python (pandas)?
  18. Find symmetric pairs quickly in numpy
  19. What is dtype(‘O’), in pandas?
  20. Pass percentiles to pandas agg function
  21. NumPy version of “Exponential weighted moving average”, equivalent to pandas.ewm().mean()
  22. Unpivot Pandas Data
  23. pandas.read_csv from string or package data
  24. Get first and second highest values in pandas columns
  25. ImportError: No module named ‘pandas.indexes’
  26. How to automatically annotate maximum value in pyplot
  27. Split Python sequence (time series/array) into subsequences with overlap
  28. Add density curve on the histogram
  29. Bin values based on ranges with pandas [duplicate]
  30. Numpy.dtype has the wrong size, try recompiling

Leave a Comment