Unicode (UTF-8) reading and writing to files in Python

by

Rather than mess with the encode and decode methods I find it easier to specify the encoding when opening the file. The io module (added in Python 2.6) provides an io.open function, which has an encoding parameter.

Use the open method from the io module.

>>>import io
>>>f = io.open("test", mode="r", encoding="utf-8")

Then after calling f’s read() function, an encoded Unicode object is returned.

>>>f.read()
u'Capit\xe1l\n\n'

Note that in Python 3, the io.open function is an alias for the built-in open function. The built-in open function only supports the encoding argument in Python 3, not Python 2.

Edit: Previously this answer recommended the codecs module. The codecs module can cause problems when mixing read() and readline(), so this answer now recommends the io module instead.

Use the open method from the codecs module.

>>>import codecs
>>>f = codecs.open("test", "r", "utf-8")

Then after calling f’s read() function, an encoded Unicode object is returned.

>>>f.read()
u'Capit\xe1l\n\n'

If you know the encoding of a file, using the codecs package is going to be much less confusing.

See http://docs.python.org/library/codecs.html#codecs.open

More Related Contents:

  1. Saving utf-8 texts with json.dumps as UTF8, not as \u escape sequence
  2. How to remove \xa0 from string in Python?
  3. u’\ufeff’ in Python string
  4. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xef in position 1
  5. Convert Unicode to ASCII without errors in Python
  6. Python & MySql: Unicode and Encoding
  7. Python and BeautifulSoup encoding issues [duplicate]
  8. How to convert a string to utf-8 in Python
  9. How do I check if a string is unicode or ascii?
  10. Writing UTF-8 String to MySQL with Python
  11. General Unicode/UTF-8 support for csv files in Python 2.6
  12. MySQL “incorrect string value” error when save unicode string in Django
  13. Why doesn’t Python recognize my utf-8 encoded source file?
  14. How to correctly parse UTF-8 encoded HTML to Unicode strings with BeautifulSoup? [duplicate]
  15. Python 3 print() function with Farsi/Arabic characters [duplicate]
  16. Python ascii utf unicode
  17. Python DictWriter writing UTF-8 encoded CSV files
  18. ElementTree and unicode
  19. Saving UTF-8 texts with json.dumps as UTF-8, not as a \u escape sequence
  20. Convert UTF-16 to UTF-8 and remove BOM?
  21. Convert “\x” escaped string into readable string in python
  22. python encoding utf-8
  23. Python – dealing with mixed-encoding files
  24. SQLite, python, unicode, and non-utf data
  25. UnicodeDecodeError when performing os.walk
  26. Bytes in a unicode Python string
  27. Python UnicodeDecodeError – Am I misunderstanding encode?
  28. How to make the python interpreter correctly handle non-ASCII characters in string operations?
  29. ValueError : I/O operation on closed file
  30. How to send utf-8 e-mail?

Leave a Comment