What is the fastest way to output large DataFrame into a CSV file?

by

Lev. Pandas has rewritten to_csv to make a big improvement in native speed. The process is now i/o bound, accounts for many subtle dtype issues, and quote cases. Here is our performance results vs. 0.10.1 (in the upcoming 0.11) release. These are in ms, lower ratio is better.

Results:
                                            t_head  t_baseline      ratio
name                                                                     
frame_to_csv2 (100k) rows                 190.5260   2244.4260     0.0849
write_csv_standard  (10k rows)             38.1940    234.2570     0.1630
frame_to_csv_mixed  (10k rows, mixed)     369.0670   1123.0412     0.3286
frame_to_csv (3k rows, wide)              112.2720    226.7549     0.4951

So Throughput for a single dtype (e.g. floats), not too wide is about 20M rows / min, here is your example from above.

In [12]: df = pd.DataFrame({'A' : np.array(np.arange(45000000),dtype="float64")}) 
In [13]: df['B'] = df['A'] + 1.0   
In [14]: df['C'] = df['A'] + 2.0
In [15]: df['D'] = df['A'] + 2.0
In [16]: %timeit -n 1 -r 1 df.to_csv('test.csv')
1 loops, best of 1: 119 s per loop

More Related Contents:

  1. When should I (not) want to use pandas apply() in my code?
  2. Does pandas iterrows have performance issues?
  3. Why does concatenation of DataFrames get exponentially slower?
  4. Replace values in a pandas series via dictionary efficiently
  5. What is the most efficient way to loop through dataframes with pandas?
  6. pandas loc vs. iloc vs. at vs. iat?
  7. Performance of Pandas apply vs np.vectorize to create new column from existing columns
  8. Pandas: peculiar performance drop for inplace rename after dropna
  9. What is the performance impact of non-unique indexes in pandas?
  10. Efficiently return the index of the first value satisfying condition in array
  11. Most efficient way to forward-fill NaN values in numpy array
  12. Fastest way to sort each row in a pandas dataframe
  13. Vectorizing Haversine distance calculation in Python
  14. Pandas pd.Series.isin performance with set versus array
  15. Vectorizing or Speeding up Fuzzywuzzy String Matching on PANDAS Column
  16. Select the max row per group – pandas performance issue
  17. Getting days since last occurence in Pandas DataFrame?
  18. Split Python sequence (time series/array) into subsequences with overlap
  19. Can I perform dynamic cumsum of rows in pandas?
  20. Stacked Bar Chart with Centered Labels
  21. Python threads all executing on a single core
  22. pandas .at versus .loc
  23. How to do “(df1 & not df2)” dataframe merge in pandas?
  24. How do I assign values based on multiple conditions for existing columns?
  25. How indexing works in Pandas?
  26. Python: How to drop a row whose particular column is empty/NaN?
  27. Discrete legend in seaborn heatmap plot
  28. Why is NaN considered as a float?
  29. Python Pandas : pandas.to_datetime() is switching day & month when day is less than 13
  30. Wide to long data transform in pandas

Leave a Comment