Reading Unicode file data with BOM chars in Python

by

There is no reason to check if a BOM exists or not, utf-8-sig manages that for you and behaves exactly as utf-8 if the BOM does not exist:

# Standard UTF-8 without BOM
>>> b'hello'.decode('utf-8')
'hello'
>>> b'hello'.decode('utf-8-sig')
'hello'

# BOM encoded UTF-8
>>> b'\xef\xbb\xbfhello'.decode('utf-8')
'\ufeffhello'
>>> b'\xef\xbb\xbfhello'.decode('utf-8-sig')
'hello'

In the example above, you can see utf-8-sig correctly decodes the given string regardless of the existence of BOM. If you think there is even a small chance that a BOM character might exist in the files you are reading, just use utf-8-sig and not worry about it

More Related Contents:

  1. What is the best way to remove accents (normalize) in a Python unicode string?
  2. How to set sys.stdout encoding in Python 3?
  3. Replace non-ASCII characters with a single space
  4. Python regex matching Unicode properties
  5. Convert Unicode to ASCII without errors in Python
  6. How to correct TypeError: Unicode-objects must be encoded before hashing?
  7. UnicodeDecodeError, invalid continuation byte
  8. How to make the python interpreter correctly handle non-ASCII characters in string operations?
  9. Get a list of all the encodings Python can encode to
  10. Writing UTF-8 String to MySQL with Python
  11. urllib2 read to Unicode
  12. ‘UCS-2’ codec can’t encode characters in position 1050-1050
  13. Why doesn’t Python recognize my utf-8 encoded source file?
  14. remove unicode emoji using re in python
  15. How do I read an image from a path with Unicode characters?
  16. How to make Django slugify work properly with Unicode strings?
  17. Python ascii utf unicode
  18. Identifier normalization: Why is the micro sign converted into the Greek letter mu?
  19. latin-1 to ascii
  20. Can’t open Unicode URL with Python
  21. Python unicode regular expression matching failing with some unicode characters -bug or mistake?
  22. python: unicode in Windows terminal, encoding used?
  23. Python 3 UnicodeDecodeError: ‘charmap’ codec can’t decode byte 0x9d
  24. How do I get str.translate to work with Unicode strings?
  25. How to encode Python 3 string using \u escape code?
  26. Python returns length of 2 for single Unicode character string
  27. SQLite, python, unicode, and non-utf data
  28. UnicodeDecodeError when performing os.walk
  29. Remove non-ASCII characters from a string using python / django
  30. Python Unicode string stored as ‘\u84b8\u6c7d\u5730’ in file, how to convert it back to Unicode?

Leave a Comment