Switching to Python 3 causing UnicodeDecodeError

by

Python 3 decodes text files when reading, encodes when writing. The default encoding is taken from locale.getpreferredencoding(False), which evidently for your setup returns 'ASCII'. See the open() function documenation:

In text mode, if encoding is not specified the encoding used is platform dependent: locale.getpreferredencoding(False) is called to get the current locale encoding.

Instead of relying on a system setting, you should open your text files using an explicit codec:

currentFile = open(filename, 'rt', encoding='latin1')

where you set the encoding parameter to match the file you are reading.

Python 3 supports UTF-8 as the default for source code.

The same applies to writing to a writeable text file; data written will be encoded, and if you rely on the system encoding you are liable to get UnicodeEncodingError exceptions unless you explicitly set a suitable codec. What codec to use when writing depends on what text you are writing and what you plan to do with the file afterward.

You may want to read up on Python 3 and Unicode in the Unicode HOWTO, which explains both about source code encoding and reading and writing Unicode data.

More Related Contents:

  1. Let JSON object accept bytes or let urlopen output strings
  2. Why doesn’t Python recognize my utf-8 encoded source file?
  3. Should I use encoding declaration in Python 3?
  4. Python 3: How to specify stdin encoding
  5. Changing the “locale preferred encoding” in Python 3 in Windows
  6. Python 3 CSV file giving UnicodeDecodeError: ‘utf-8’ codec can’t decode byte error when I print
  7. Swapping variable places inside if throws an error
  8. What is the best way to remove accents (normalize) in a Python unicode string?
  9. How to install python3 version of package via pip on Ubuntu?
  10. “for line in…” results in UnicodeDecodeError: ‘utf-8’ codec can’t decode byte
  11. How to postpone/defer the evaluation of f-strings?
  12. Why does Python assignment not return a value?
  13. What’s the difference between “pip install” and “python -m pip install”?
  14. How to write binary data to stdout in python 3?
  15. Difference between python3 and python3m executables
  16. Python 3 replacement for deprecated compiler.ast flatten function
  17. exec() and variable scope [duplicate]
  18. warning about too many open figures
  19. What does Python optimization (-O or PYTHONOPTIMIZE) do?
  20. How to remove or hide x-axis labels from a seaborn / matplotlib plot
  21. Clarification on the Decimal type in Python
  22. Remove NaN ‘Cells’ without dropping the entire ROW (Pandas,Python3)
  23. How can I activate a virtualenv in Linux?
  24. Number rows within group in increasing order in a pandas dataframe
  25. append dictionary to data frame
  26. Switch between python 2.7 and python 3.5 on Mac OS X
  27. How to use sys.exit() in Python
  28. Is there a way to use asyncio.Queue in multiple threads?
  29. Add custom form fields that are not part of the model (Django)
  30. Writing to a new file if it doesn’t exist, and appending to a file if it does

Leave a Comment