Pandas: peculiar performance drop for inplace rename after dropna

by

This is a copy of the explanation on github.

There is no guarantee that an inplace operation is actually faster. Often they are actually the same operation that works on a copy, but the top-level reference is reassigned.

The reason for the difference in performance in this case is as follows.

The (df1-df2).dropna() call creates a slice of the dataframe. When you apply a new operation, this triggers a SettingWithCopy check because it could be a copy (but often is not).

This check must perform a garbage collection to wipe out some cache references to see if it’s a copy. Unfortunately python syntax makes this unavoidable.

You can not have this happen, by simply making a copy first.

df = (df1-df2).dropna().copy()

followed by an inplace operation will be as performant as before.

My personal opinion: I never use in-place operations. The syntax is harder to read and it does not offer any advantages.

More Related Contents:

  1. When should I (not) want to use pandas apply() in my code?
  2. Does pandas iterrows have performance issues?
  3. Why does concatenation of DataFrames get exponentially slower?
  4. Understanding inplace=True in pandas
  5. Replace values in a pandas series via dictionary efficiently
  6. What is the most efficient way to loop through dataframes with pandas?
  7. pandas loc vs. iloc vs. at vs. iat?
  8. Performance of Pandas apply vs np.vectorize to create new column from existing columns
  9. What is the performance impact of non-unique indexes in pandas?
  10. Efficiently return the index of the first value satisfying condition in array
  11. Most efficient way to forward-fill NaN values in numpy array
  12. Fastest way to sort each row in a pandas dataframe
  13. Vectorizing Haversine distance calculation in Python
  14. Pandas pd.Series.isin performance with set versus array
  15. Vectorizing or Speeding up Fuzzywuzzy String Matching on PANDAS Column
  16. Pandas: drop columns with all NaN’s
  17. What is the fastest way to output large DataFrame into a CSV file?
  18. Select the max row per group – pandas performance issue
  19. Getting days since last occurence in Pandas DataFrame?
  20. Split Python sequence (time series/array) into subsequences with overlap
  21. Can I perform dynamic cumsum of rows in pandas?
  22. Difference between map, applymap and apply methods in Pandas
  23. How to replace text in a string column of a Pandas dataframe?
  24. Pandas dataframe get first row of each group
  25. Remove index name in pandas
  26. Moving Average Pandas
  27. Filter string data based on its string length
  28. Matplotlib showing x-tick labels overlapping
  29. Pandas custom function to find whether it is the 1st, 2nd etc Monday, Tuesday, etc – all suggestions welcome
  30. numpy faster than numba and cython , how to improve numba code

Leave a Comment