Setting the correct encoding when piping stdout in Python

by

Your code works when run in an script because Python encodes the output to whatever encoding your terminal application is using. If you are piping you must encode it yourself.

A rule of thumb is: Always use Unicode internally. Decode what you receive, and encode what you send.

# -*- coding: utf-8 -*-
print u"åäö".encode('utf-8')

Another didactic example is a Python program to convert between ISO-8859-1 and UTF-8, making everything uppercase in between.

import sys
for line in sys.stdin:
    # Decode what you receive:
    line = line.decode('iso8859-1')

    # Work with Unicode internally:
    line = line.upper()

    # Encode what you send:
    line = line.encode('utf-8')
    sys.stdout.write(line)

Setting the system default encoding is a bad idea, because some modules and libraries you use can rely on the fact it is ASCII. Don’t do it.

More Related Contents:

  1. Why should we NOT use sys.setdefaultencoding(“utf-8”) in a py script?
  2. Why does Python print unicode characters when the default encoding is ASCII?
  3. Dangers of sys.setdefaultencoding(‘utf-8’)
  4. Python ascii utf unicode
  5. How does Python 2 compare string and int? Why do lists compare as greater than numbers, and tuples greater than lists?
  6. What is the best way to remove accents (normalize) in a Python unicode string?
  7. Changing default encoding of Python?
  8. Taking multiple integers on the same line as input from the user in python
  9. Match text between two strings with regular expression
  10. What exactly do “u” and “r” string prefixes do, and what are raw string literals?
  11. What is the difference between dict.items() and dict.iteritems() in Python2?
  12. How to open every file in a folder
  13. Why is ” > 0 True in Python 2?
  14. Intercepting stdout of a subprocess while it is running
  15. How to use a multiprocessing.Manager()?
  16. Converting “yield from” statement to Python 2.7 code
  17. Command executed with Paramiko does not produce any output
  18. How to print a percentage value in python?
  19. How do I check if stdin has some data?
  20. How to handle response encoding from urllib.request.urlopen() , to avoid TypeError: can’t use a string pattern on a bytes-like object
  21. Safest way to convert float to integer in python?
  22. Make python3 as my default python on Mac
  23. How to select a directory and store the location using tkinter in Python
  24. Command-line input causes SyntaxError
  25. python: unicode in Windows terminal, encoding used?
  26. Switch between python 2.7 and python 3.5 on Mac OS X
  27. How to Print “Pretty” String Output in Python
  28. Python module to enable ANSI colors for stdout on Windows?
  29. What arguments does Python sort() function have?
  30. How can I capture the stdout output of a child process?

Leave a Comment