Python 3 CSV file giving UnicodeDecodeError: ‘utf-8’ codec can’t decode byte error when I print

by

We know the file contains the byte b'\x96' since it is mentioned in the error message:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 7386: invalid start byte

Now we can write a little script to find out if there are any encodings where b'\x96' decodes to ñ:

import pkgutil
import encodings
import os

def all_encodings():
    modnames = set([modname for importer, modname, ispkg in pkgutil.walk_packages(
        path=[os.path.dirname(encodings.__file__)], prefix='')])
    aliases = set(encodings.aliases.aliases.values())
    return modnames.union(aliases)

text = b'\x96'
for enc in all_encodings():
    try:
        msg = text.decode(enc)
    except Exception:
        continue
    if msg == 'ñ':
        print('Decoding {t} with {enc} is {m}'.format(t=text, enc=enc, m=msg))

which yields

Decoding b'\x96' with mac_roman is ñ
Decoding b'\x96' with mac_farsi is ñ
Decoding b'\x96' with mac_croatian is ñ
Decoding b'\x96' with mac_arabic is ñ
Decoding b'\x96' with mac_romanian is ñ
Decoding b'\x96' with mac_iceland is ñ
Decoding b'\x96' with mac_turkish is ñ

Therefore, try changing

with open('my_file.csv', 'r', newline="") as csvfile:

to one of those encodings, such as:

with open('my_file.csv', 'r', encoding='mac_roman', newline="") as csvfile:

More Related Contents:

  1. How to write UTF-8 in a CSV file
  2. Why doesn’t Python recognize my utf-8 encoded source file?
  3. ‘utf-8’ codec can’t decode byte 0xa0 in position 4276: invalid start byte
  4. Should I use encoding declaration in Python 3?
  5. Changing the “locale preferred encoding” in Python 3 in Windows
  6. A simple maths operation on a CSV column using python [closed]
  7. Changing default encoding of Python?
  8. Reading a UTF8 CSV file with Python
  9. How to add a new column to a CSV file?
  10. csv.Error: iterator should return strings, not bytes
  11. How to convert a file to utf-8 in Python?
  12. printing UTF-8 in Python 3 using Sublime Text 3
  13. How to print UTF-8 encoded text to the console in Python < 3?
  14. Python 3 print() function with Farsi/Arabic characters [duplicate]
  15. portable way to write csv file in python 2 or python 3
  16. Python 3: How to specify stdin encoding
  17. Python ascii utf unicode
  18. Loading UTF-8 file in Python 3 using numpy.genfromtxt
  19. python pandas read_csv delimiter in column data
  20. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xc3 in position 23: ordinal not in range(128)
  21. Why does this Python program send empty emails when I encode it with utf-8? [duplicate]
  22. Switching to Python 3 causing UnicodeDecodeError
  23. UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x8b in position 1: invalid start byte, while reading csv file in pandas
  24. Convert “\x” escaped string into readable string in python
  25. Pandas ParserError EOF character when reading multiple csv files to HDF5
  26. UnicodeDecodeError in Python when reading a file, how to ignore the error and jump to the next line?
  27. Read .csv file from URL into Python 3.x – _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
  28. Python – dealing with mixed-encoding files
  29. How to write List of lists in csv file in python [duplicate]
  30. Python CGI – UTF-8 doesn’t work

Leave a Comment