Why does concatenation of DataFrames get exponentially slower?

Never call DataFrame.append or pd.concat inside a for-loop. It leads to quadratic copying. pd.concat returns a new DataFrame. Space has to be allocated for the new DataFrame, and data from the old DataFrames have to be copied into the new DataFrame. Consider the amount of copying required by this line inside the for-loop (assuming each … Read more