Using numpy.genfromtxt to read a csv file with strings containing commas

by

You can use pandas (the becoming default library for working with dataframes (heterogeneous data) in scientific python) for this. It’s read_csv can handle this. From the docs:

quotechar : string

The character to used to denote the start and end of a quoted item. Quoted items 
can include the delimiter and it will be ignored.

The default value is ". An example:

In [1]: import pandas as pd

In [2]: from StringIO import StringIO

In [3]: s="""year, city, value
   ...: 2012, "Louisville KY", 3.5
   ...: 2011, "Lexington, KY", 4.0"""

In [4]: pd.read_csv(StringIO(s), quotechar=""", skipinitialspace=True)
Out[4]:
   year           city  value
0  2012  Louisville KY    3.5
1  2011  Lexington, KY    4.0

The trick here is that you also have to use skipinitialspace=True to deal with the spaces after the comma-delimiter.

Apart from a powerful csv reader, I can also strongly advice to use pandas with the heterogeneous data you have (the example output in numpy you give are all strings, although you could use structured arrays).

More Related Contents:

  1. selecting across multiple columns with python pandas?
  2. Precision lost while using read_csv in pandas
  3. pandas.read_csv from string or package data
  4. Load CSV to Pandas MultiIndex DataFrame
  5. Bin values based on ranges with pandas [duplicate]
  6. Pandas conditional creation of a series/dataframe column
  7. How do I read CSV data into a record array in NumPy?
  8. Writing a pandas DataFrame to CSV file
  9. Fast Haversine Approximation (Python/Pandas)
  10. Pandas Timedelta in Days
  11. Conditionally fill column values based on another columns value in pandas
  12. Calculate weighted average using a pandas/dataframe
  13. How to remove blanks/NA’s from dataframe and shift the values up
  14. Ambiguity in Pandas Dataframe / Numpy Array “axis” definition
  15. Vectorized look-up of values in Pandas dataframe
  16. numpy.genfromtxt produces array of what looks like tuples, not a 2D array—why?
  17. Pandas: How to create a datetime object from Week and Year?
  18. python pandas read_csv delimiter in column data
  19. Numpy isnan() fails on an array of floats (from pandas dataframe apply)
  20. plotting a histogram on a Log scale with Matplotlib
  21. module ‘pandas’ has no attribute ‘read_csv
  22. Conditional row read of csv in pandas
  23. Calculate summary statistics of columns in dataframe
  24. Extract csv file specific columns to list in Python
  25. double quoted elements in csv cant read with pandas
  26. Resample time series in pandas to a weekly interval
  27. How to use numpy to get the cumulative count by unique values in linear time?
  28. UnicodeDecodeError when reading CSV file in Pandas
  29. How to return a csv file/Pandas DataFrame in JSON format using FastAPI?
  30. ValueError: x and y must be the same size

Leave a Comment