Write to UTF-8 file in Python

by

I believe the problem is that codecs.BOM_UTF8 is a byte string, not a Unicode string. I suspect the file handler is trying to guess what you really mean based on “I’m meant to be writing Unicode as UTF-8-encoded text, but you’ve given me a byte string!”

Try writing the Unicode string for the byte order mark (i.e. Unicode U+FEFF) directly, so that the file just encodes that as UTF-8:

import codecs

file = codecs.open("lol", "w", "utf-8")
file.write(u'\ufeff')
file.close()

(That seems to give the right answer – a file with bytes EF BB BF.)

EDIT: S. Lott’s suggestion of using “utf-8-sig” as the encoding is a better one than explicitly writing the BOM yourself, but I’ll leave this answer here as it explains what was going wrong before.

More Related Contents:

  1. Working with UTF-8 encoding in Python source [duplicate]
  2. Reading a UTF8 CSV file with Python
  3. Convert Unicode to ASCII without errors in Python
  4. Convert UTF-8 with BOM to UTF-8 with no BOM in Python
  5. python 3 open() default encoding
  6. What encoding does open() use by default?
  7. UTF-8 HTML and CSS files with BOM (and how to remove the BOM with Python)
  8. Bytes in a unicode Python string
  9. Best way to convert string to bytes in Python 3?
  10. Unicode (UTF-8) reading and writing to files in Python
  11. Python & MySql: Unicode and Encoding
  12. A good way to get the charset/encoding of an HTTP response in Python
  13. Python and BeautifulSoup encoding issues [duplicate]
  14. Writing UTF-8 String to MySQL with Python
  15. printing UTF-8 in Python 3 using Sublime Text 3
  16. The smtplib.server.sendmail function in python raises UnicodeEncodeError: ‘ascii’ codec can’t encode character
  17. MySQL “incorrect string value” error when save unicode string in Django
  18. How to print UTF-8 encoded text to the console in Python < 3?
  19. How to display utf-8 in windows console
  20. Remove non-ASCII characters from pandas column
  21. How to unquote a urlencoded unicode string in python?
  22. Python reading from a file and saving to utf-8
  23. Python and character normalization
  24. Python ascii utf unicode
  25. Loading UTF-8 file in Python 3 using numpy.genfromtxt
  26. What’s the difference between UTF-8 and UTF-8 with BOM?
  27. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xc3 in position 23: ordinal not in range(128)
  28. UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\u2013′ in position 3 2: ordinal not in range(128)
  29. Python 3 CSV file giving UnicodeDecodeError: ‘utf-8’ codec can’t decode byte error when I print
  30. Pandas df.to_csv(“file.csv” encode=”utf-8″) still gives trash characters for minus sign

Leave a Comment