Python reading from a file and saving to utf-8

by

Process text to and from Unicode at the I/O boundaries of your program using open with the encoding parameter. Make sure to use the (hopefully documented) encoding of the file being read. The default encoding varies by OS (specifically, locale.getpreferredencoding(False) is the encoding used), so I recommend always explicitly using the encoding parameter for portability and clarity (Python 3 syntax below):

with open(filename, 'r', encoding='utf8') as f:
    text = f.read()

# process Unicode text

with open(filename, 'w', encoding='utf8') as f:
    f.write(text)

If still using Python 2 or for Python 2/3 compatibility, the io module implements open with the same semantics as Python 3’s open and exists in both versions:

import io
with io.open(filename, 'r', encoding='utf8') as f:
    text = f.read()

# process Unicode text

with io.open(filename, 'w', encoding='utf8') as f:
    f.write(text)

More Related Contents:

  1. How to remove \xa0 from string in Python?
  2. How to convert a string to utf-8 in Python
  3. UnicodeEncodeError: ‘ascii’ codec can’t encode characters in position 0-5: ordinal not in range(128) [duplicate]
  4. Creating multiple python dictionaries from a file
  5. Why doesn’t Python run my bash code?
  6. What is the result of % in Python?
  7. u’\ufeff’ in Python string
  8. How to install PIL with pip on Mac OS?
  9. sklearn error ValueError: Input contains NaN, infinity or a value too large for dtype(‘float64’)
  10. ImportError: numpy.core.multiarray failed to import
  11. How to get PID by process name?
  12. Display realtime output of a subprocess in a tkinter widget
  13. How to install xgboost package in python (windows platform)?
  14. How to bind a click event to a Canvas in Tkinter? [closed]
  15. Matplotlib issue on OS X (“ImportError: cannot import name _thread”)
  16. How to use __setattr__ correctly, avoiding infinite recursion
  17. How to get selectors with dynamic part inside using Selenium with Python?
  18. Should I use encoding declaration in Python 3?
  19. Is it good practice to use `import __main__`?
  20. Type hinting in Python 2
  21. Script skips second for loop when reading a file
  22. Why does ENcoding a string result in a DEcoding error (UnicodeDecodeError)?
  23. Python – Homework – Converting Any Base to Any Base
  24. How to solve import error for pandas?
  25. What does “table” in the string.translate function mean?
  26. Why does dict(k=4, z=2).update(dict(l=1)) return None in Python? [duplicate]
  27. How do I make matplotlib open a box for user comments?
  28. How to create a new text file using Python
  29. Elegant way to perform tuple arithmetic
  30. ‘is’ operator behaves unexpectedly with floats

Leave a Comment