Convert UTF-16 to UTF-8 and remove BOM?

by

This is the difference between UTF-16LE and UTF-16

  • UTF-16LE is little endian without a BOM
  • UTF-16 is big or little endian with a BOM

So when you use UTF-16LE, the BOM is just part of the text. Use UTF-16 instead, so the BOM is automatically removed. The reason UTF-16LE and UTF-16BE exist is so people can carry around “properly-encoded” text without BOMs, which does not apply to you.

Note what happens when you encode using one encoding and decode using the other. (UTF-16 automatically detects UTF-16LE sometimes, not always.)

>>> u'Hello, world'.encode('UTF-16LE')
'H\x00e\x00l\x00l\x00o\x00,\x00 \x00w\x00o\x00r\x00l\x00d\x00'
>>> u'Hello, world'.encode('UTF-16')
'\xff\xfeH\x00e\x00l\x00l\x00o\x00,\x00 \x00w\x00o\x00r\x00l\x00d\x00'
 ^^^^^^^^ (BOM)

>>> u'Hello, world'.encode('UTF-16LE').decode('UTF-16')
u'Hello, world'
>>> u'Hello, world'.encode('UTF-16').decode('UTF-16LE')
u'\ufeffHello, world'
    ^^^^ (BOM)

Or you can do this at the shell:

for x in * ; do iconv -f UTF-16 -t UTF-8 <"$x" | dos2unix >"$x.tmp" && mv "$x.tmp" "$x"; done

More Related Contents:

  1. UnicodeDecodeError when performing os.walk
  2. Saving utf-8 texts with json.dumps as UTF8, not as \u escape sequence
  3. How to remove \xa0 from string in Python?
  4. u’\ufeff’ in Python string
  5. Unicode (UTF-8) reading and writing to files in Python
  6. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xef in position 1
  7. Convert Unicode to ASCII without errors in Python
  8. Convert UTF-8 with BOM to UTF-8 with no BOM in Python
  9. Python & MySql: Unicode and Encoding
  10. Python and BeautifulSoup encoding issues [duplicate]
  11. How to convert a string to utf-8 in Python
  12. How do I check if a string is unicode or ascii?
  13. Writing UTF-8 String to MySQL with Python
  14. General Unicode/UTF-8 support for csv files in Python 2.6
  15. MySQL “incorrect string value” error when save unicode string in Django
  16. Why doesn’t Python recognize my utf-8 encoded source file?
  17. How to correctly parse UTF-8 encoded HTML to Unicode strings with BeautifulSoup? [duplicate]
  18. Python 3 print() function with Farsi/Arabic characters [duplicate]
  19. Python ascii utf unicode
  20. Python DictWriter writing UTF-8 encoded CSV files
  21. ElementTree and unicode
  22. Saving UTF-8 texts with json.dumps as UTF-8, not as a \u escape sequence
  23. Convert “\x” escaped string into readable string in python
  24. python encoding utf-8
  25. Python – dealing with mixed-encoding files
  26. SQLite, python, unicode, and non-utf data
  27. Bytes in a unicode Python string
  28. Changing default encoding of Python?
  29. How do I specify a range of unicode characters
  30. Displaying unicode symbols using pygame

Leave a Comment