pandas read_csv and filter columns with usecols

by

The solution lies in understanding these two keyword arguments:

  • names is only necessary when there is no header row in your file and you want to specify other arguments (such as usecols) using column names rather than integer indices.
  • usecols is supposed to provide a filter before reading the whole DataFrame into memory; if used properly, there should never be a need to delete columns after reading.

So because you have a header row, passing header=0 is sufficient and additionally passing names appears to be confusing pd.read_csv.

Removing names from the second call gives the desired output:

import pandas as pd
from StringIO import StringIO

csv = r"""dummy,date,loc,x
bar,20090101,a,1
bar,20090102,a,3
bar,20090103,a,5
bar,20090101,b,1
bar,20090102,b,3
bar,20090103,b,5"""

df = pd.read_csv(StringIO(csv),
        header=0,
        index_col=["date", "loc"], 
        usecols=["date", "loc", "x"],
        parse_dates=["date"])

Which gives us:

                x
date       loc
2009-01-01 a    1
2009-01-02 a    3
2009-01-03 a    5
2009-01-01 b    1
2009-01-02 b    3
2009-01-03 b    5

More Related Contents:

  1. Create Pandas DataFrame from a string
  2. How do I read a large csv file with pandas?
  3. How to add pandas data to an existing csv file?
  4. Import CSV file as a pandas DataFrame
  5. Keep only date part when using pandas.to_datetime
  6. What is the fastest way to upload a big csv file in notebook to work with python pandas?
  7. Pandas split column into multiple columns by comma
  8. Convert commas decimal separators to dots within a Dataframe
  9. Pandas read csv file with float values results in weird rounding and decimal digits
  10. How to efficiently handle European decimal separators using the pandas read_csv function?
  11. Python Pandas read_csv skip rows but keep header
  12. Key error when selecting columns in pandas dataframe after read_csv
  13. Precision lost while using read_csv in pandas
  14. Python Pandas: How to read only first n rows of CSV files in?
  15. Prevent pandas read_csv treating first row as header of column names
  16. Pandas – Strip white space
  17. Filter string data based on its string length
  18. Why is pandas.to_datetime slow for non standard time format such as ‘2014/12/31’
  19. UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x96 in position 35: invalid start byte
  20. How to avoid pandas creating an index in a saved csv
  21. Pandas – write Multiindex rows with to_csv
  22. How to add header row to a pandas DataFrame
  23. UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x8b in position 1: invalid start byte, while reading csv file in pandas
  24. Pandas ParserError EOF character when reading multiple csv files to HDF5
  25. Python Pandas – Read csv file containing multiple tables
  26. Is there a way to speed up handling large CSVs and dataframes in python?
  27. Load CSV to Pandas MultiIndex DataFrame
  28. pandas read csv with extra commas in column
  29. How to upload a CSV file in FastAPI and convert it into a Pandas Dataframe?
  30. pandas.read_csv FileNotFoundError: File b’\xe2\x80\xaa’ despite correct path

Leave a Comment