Why does concatenation of DataFrames get exponentially slower?

by

Never call DataFrame.append or pd.concat inside a for-loop. It leads to quadratic copying.

pd.concat returns a new DataFrame. Space has to be allocated for the new
DataFrame, and data from the old DataFrames have to be copied into the new
DataFrame. Consider the amount of copying required by this line inside the for-loop (assuming each x has size 1):

super_x = pd.concat([super_x, x], axis=0)

| iteration | size of old super_x | size of x | copying required |
|         0 |                   0 |         1 |                1 |
|         1 |                   1 |         1 |                2 |
|         2 |                   2 |         1 |                3 |
|       ... |                     |           |                  |
|       N-1 |                 N-1 |         1 |                N |

1 + 2 + 3 + ... + N = N(N+1)/2. So there is O(N**2) copies required to
complete the loop.

Now consider

super_x = []
for i, df_chunk in enumerate(df_list):
    [x, y] = preprocess_data(df_chunk)
    super_x.append(x)
super_x = pd.concat(super_x, axis=0)

Appending to a list is an O(1) operation and does not require copying. Now
there is a single call to pd.concat after the loop is done. This call to
pd.concat requires N copies to be made, since super_x contains N
DataFrames of size 1. So when constructed this way, super_x requires O(N)
copies.

More Related Contents:

  1. Is doing multiplication multiple times or assigning to a variable faster?
  2. Pandas Merging 101
  3. When should I (not) want to use pandas apply() in my code?
  4. Import multiple csv files into pandas and concatenate into one DataFrame
  5. Does pandas iterrows have performance issues?
  6. Replace values in a pandas series via dictionary efficiently
  7. What is the most efficient way to loop through dataframes with pandas?
  8. pandas loc vs. iloc vs. at vs. iat?
  9. Performance of Pandas apply vs np.vectorize to create new column from existing columns
  10. Pandas: peculiar performance drop for inplace rename after dropna
  11. What is the performance impact of non-unique indexes in pandas?
  12. Efficiently return the index of the first value satisfying condition in array
  13. Import multiple excel files into python pandas and concatenate them into one dataframe
  14. Most efficient way to forward-fill NaN values in numpy array
  15. pandas concat generates nan values
  16. Fastest way to sort each row in a pandas dataframe
  17. Vectorizing Haversine distance calculation in Python
  18. Pandas pd.Series.isin performance with set versus array
  19. Vectorizing or Speeding up Fuzzywuzzy String Matching on PANDAS Column
  20. Pandas concat yields ValueError: Plan shapes are not aligned
  21. What is the fastest way to output large DataFrame into a CSV file?
  22. Select the max row per group – pandas performance issue
  23. Getting days since last occurence in Pandas DataFrame?
  24. How to convert DataFrame.append() to pandas.concat()?
  25. Split Python sequence (time series/array) into subsequences with overlap
  26. Can I perform dynamic cumsum of rows in pandas?
  27. Coalesce values from 2 columns into a single column in a pandas dataframe
  28. Converting pandas column of comma-separated strings into dummy variables
  29. Check if rows in one dataframe exist in another dataframe
  30. Add column of empty lists to DataFrame

Leave a Comment