Python 3 chokes on CP-1252/ANSI reading

by

Position 0x81 is unassigned in Windows-1252 (aka cp1252). It is assigned to U+0081 HIGH OCTET PRESET (HOP) control character in Latin-1 (aka ISO 8859-1). I can reproduce your error in Python 3.1 like this:

>>> b'\x81'.decode('cp1252')
Traceback (most recent call last):
  ...
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 0: character maps to <undefined>

or with an actual file:

>>> open('test.txt', 'wb').write(b'\x81\n')
2
>>> open('test.txt').read()
Traceback (most recent call last):
  ...
UnicodeDecodeError: 'utf8' codec can't decode byte 0x81 in position 0: unexpected code byte

Now to treat this file as Latin-1 you pass the encoding argument, like codeape suggested:

>>> open('test.txt', encoding='latin-1').read()
'\x81\n'

Beware that there are differences between Windows-1257 and Latin-1 encodings, e.g. Latin-1 doesn’t have “smart quotes”. If the file you’re processing is a text file, ask yourself what that \x81 is doing in it.

More Related Contents:

  1. What is the best way to remove accents (normalize) in a Python unicode string?
  2. UnicodeDecodeError: ‘charmap’ codec can’t decode byte X in position Y: character maps to
  3. How to set sys.stdout encoding in Python 3?
  4. “Unicode Error “unicodeescape” codec can’t decode bytes… Cannot open text files in Python 3 [duplicate]
  5. How can I convert surrogate pairs to normal string in Python?
  6. How to correct TypeError: Unicode-objects must be encoded before hashing?
  7. Normalizing Unicode
  8. Facebook JSON badly encoded
  9. NameError: global name ‘unicode’ is not defined – in Python 3
  10. Why doesn’t Python recognize my utf-8 encoded source file?
  11. Python 3 print() function with Farsi/Arabic characters [duplicate]
  12. How to make unicode string with python3
  13. Python 3: How to specify stdin encoding
  14. Identifier normalization: Why is the micro sign converted into the Greek letter mu?
  15. requests.get returns 403 while the same url works in browser
  16. How to encode Python 3 string using \u escape code?
  17. What is internal representation of string in Python 3.x
  18. Set encoding in Python 3 CGI scripts
  19. How to deal with unicode string in URL in python3?
  20. How to make print() output UTF-8 in Python 3.0?
  21. What’s the difference between “pip install” and “python -m pip install”?
  22. Two variables in Python have same id, but not lists or tuples
  23. How to write binary data to stdout in python 3?
  24. How unique is Python’s id()?
  25. Python 3 replacement for deprecated compiler.ast flatten function
  26. exec() and variable scope [duplicate]
  27. warning about too many open figures
  28. How to unquote a urlencoded unicode string in python?
  29. What does Python optimization (-O or PYTHONOPTIMIZE) do?
  30. How to properly use the “choices” field option in Django

Leave a Comment