Fastest way to parse large CSV files in Pandas

by

As @chrisb said, pandas’ read_csv is probably faster than csv.reader/numpy.genfromtxt/loadtxt. I don’t think you will find something better to parse the csv (as a note, read_csv is not a ‘pure python’ solution, as the CSV parser is implemented in C).

But, if you have to load/query the data often, a solution would be to parse the CSV only once and then store it in another format, eg HDF5. You can use pandas (with PyTables in background) to query that efficiently (docs).
See here for a comparison of the io performance of HDF5, csv and SQL with pandas: http://pandas.pydata.org/pandas-docs/stable/io.html#performance-considerations

And a possibly relevant other question: “Large data” work flows using pandas

More Related Contents:

  1. How to provide a reproducible copy of your DataFrame with to_clipboard()
  2. Combine two columns of text in pandas dataframe
  3. How to apply a function to two columns of Pandas dataframe
  4. How to convert an XML file to nice pandas dataframe?
  5. Dynamically evaluate an expression from a formula in Pandas
  6. How to get rid of “Unnamed: 0” column in a pandas DataFrame read in from CSV file?
  7. Load data from txt with pandas
  8. Add Leading Zeros to Strings in Pandas Dataframe
  9. Progress indicator during pandas operations
  10. Coloring Cells in Pandas
  11. How to import data from mongodb to pandas?
  12. How to group a pandas dataframe by a defined time interval?
  13. How to sort a Pandas DataFrame by index?
  14. custom matplotlib plot : chess board like table with colored cells
  15. How do you plot a vertical line on a time series plot in Pandas?
  16. pandas applying regex to replace values
  17. why does pandas rolling use single dimension ndarray
  18. pandas two dataframe cross join [duplicate]
  19. Bin pandas dataframe by every X rows
  20. Redefining the Index in a Pandas DataFrame object
  21. pandas .plot() x-axis tick frequency — how can I show more ticks?
  22. module ‘pandas’ has no attribute ‘rolling_mean’
  23. Group dataframe and get sum AND count?
  24. pandas apply function that returns multiple values to rows in pandas dataframe
  25. Improve Pandas Merge performance
  26. Logarithmic returns in pandas dataframe
  27. Jupyter python3 notebook cannot recognize pandas
  28. Python: reduce precision pandas timestamp dataframe
  29. python: update dataframe to existing excel sheet without overwriting contents on the same sheet and other sheets
  30. How to select the last column of dataframe

Leave a Comment