Reading a UTF8 CSV file with Python

by

The .encode method gets applied to a Unicode string to make a byte-string; but you’re calling it on a byte-string instead… the wrong way ’round! Look at the codecs module in the standard library and codecs.open in particular for better general solutions for reading UTF-8 encoded text files. However, for the csv module in particular, you need to pass in utf-8 data, and that’s what you’re already getting, so your code can be much simpler:

import csv

def unicode_csv_reader(utf8_data, dialect=csv.excel, **kwargs):
    csv_reader = csv.reader(utf8_data, dialect=dialect, **kwargs)
    for row in csv_reader:
        yield [unicode(cell, 'utf-8') for cell in row]

filename="da.csv"
reader = unicode_csv_reader(open(filename))
for field1, field2, field3 in reader:
  print field1, field2, field3

PS: if it turns out that your input data is NOT in utf-8, but e.g. in ISO-8859-1, then you do need a “transcoding” (if you’re keen on using utf-8 at the csv module level), of the form line.decode('whateverweirdcodec').encode('utf-8') — but probably you can just use the name of your existing encoding in the yield line in my code above, instead of 'utf-8', as csv is actually going to be just fine with ISO-8859-* encoded bytestrings.

More Related Contents:

  1. Working with UTF-8 encoding in Python source [duplicate]
  2. Convert Unicode to ASCII without errors in Python
  3. Write to UTF-8 file in Python
  4. General Unicode/UTF-8 support for csv files in Python 2.6
  5. How to write UTF-8 in a CSV file
  6. ‘utf-8’ codec can’t decode byte 0xa0 in position 4276: invalid start byte
  7. Loading UTF-8 file in Python 3 using numpy.genfromtxt
  8. Python DictWriter writing UTF-8 encoded CSV files
  9. python 3 open() default encoding
  10. What encoding does open() use by default?
  11. Python 3 CSV file giving UnicodeDecodeError: ‘utf-8’ codec can’t decode byte error when I print
  12. Pandas df.to_csv(“file.csv” encode=”utf-8″) still gives trash characters for minus sign
  13. Bytes in a unicode Python string
  14. How do I create a python code that distinguishes between the capital of a state and the state that the capital has?
  15. How do I read a large csv file with pandas?
  16. Unicode (UTF-8) reading and writing to files in Python
  17. Python csv string to array
  18. “Line contains NULL byte” in CSV reader (Python)
  19. Python and BeautifulSoup encoding issues [duplicate]
  20. Reading data from a CSV file in Python
  21. How to efficiently handle European decimal separators using the pandas read_csv function?
  22. Python Pandas read_csv skip rows but keep header
  23. Pandas – Strip white space
  24. Filter string data based on its string length
  25. Why is pandas.to_datetime slow for non standard time format such as ‘2014/12/31’
  26. Python Pandas – Read csv file containing multiple tables
  27. Find all CSV files in a directory using Python
  28. how to specify the datetime format in read_csv
  29. How to upload a CSV file in FastAPI and convert it into JSON?
  30. Convert nested JSON to CSV file in Python

Leave a Comment