How can I filter lines on load in Pandas read_csv function?

by

There isn’t an option to filter the rows before the CSV file is loaded into a pandas object.

You can either load the file and then filter using df[df['field'] > constant], or if you have a very large file and you are worried about memory running out, then use an iterator and apply the filter as you concatenate chunks of your file e.g.:

import pandas as pd
iter_csv = pd.read_csv('file.csv', iterator=True, chunksize=1000)
df = pd.concat([chunk[chunk['field'] > constant] for chunk in iter_csv])

You can vary the chunksize to suit your available memory. See here for more details.

More Related Contents:

  1. How can I pivot a dataframe?
  2. GroupBy pandas DataFrame and select most common value
  3. String concatenation of two pandas columns
  4. Reshape wide to long in pandas
  5. Efficiently return the index of the first value satisfying condition in array
  6. PyInstaller and Pandas
  7. Converting between datetime and Pandas Timestamp objects
  8. Pandas DataFrame column to list [duplicate]
  9. Bulk Insert A Pandas DataFrame Using SQLAlchemy
  10. How to plot multiple pandas columns
  11. Shift NaNs to the end of their respective rows
  12. multi index plotting
  13. Pandas how can ‘replace’ work after ‘loc’?
  14. How to slice a pandas DataFrame by position?
  15. Pandas Dataframe: split column into multiple columns, right-align inconsistent cell entries
  16. Pandas in AWS lambda gives numpy error
  17. Writing a Python Pandas DataFrame to Word document
  18. Select row from a DataFrame based on the type of the object(i.e. str)
  19. Selecting Data between Specific hours in a pandas dataframe
  20. How to annotate a seaborn barplot with the aggregated value
  21. Add numpy array as column to Pandas data frame
  22. Python: How to drop a row whose particular column is empty/NaN?
  23. Discrete legend in seaborn heatmap plot
  24. How to use ast.literal_eval in a pandas dataframe and handle exceptions
  25. Why is NaN considered as a float?
  26. Clean way to convert quarterly periods to datetime in pandas
  27. Combine Pandas data frame column values into new column
  28. Python Pandas : pandas.to_datetime() is switching day & month when day is less than 13
  29. Wide to long data transform in pandas
  30. Is it possible to add a string as a legend item

Leave a Comment