Python ascii utf unicode

by

You already have the value. Python simply tries to make debugging easier by giving you a representation that is ASCII friendly. Echoing values in the interpreter gives you the result of calling repr() on the result.

In other words, you are confusing the representation of the value with the value itself. The representation is designed to be safely copied and pasted around, without worry about how other systems might handle non-ASCII codepoints. As such the Python string literal syntax is used, with any non-printable and non-ASCII characters replaced by \xhh and \uhhhh escape sequences. Pasting those strings back into a Python string or interactive Python session will reproduce the exact same value.

As such ü has been replaced by \xfc, because that’s the Unicode codepoint for the U+00FC LATIN SMALL LETTER U WITH DIAERESIS codepoint.

If your terminal is configured correctly, you can just use print and Python will encode the Unicode value to your terminal codec, resulting in your terminal display giving you the non-ASCII glyphs:

>>> u'Fortuna Düsseldorf'
u'Fortuna D\xfcsseldorf'
>>> print u'Fortuna Düsseldorf'
Fortuna Düsseldorf

If your terminal is configured for UTF-8, you can also write the UTF-8 bytes directly to your terminal, after encoding explicitly:

>>> u'Fortuna Düsseldorf'.encode('utf8')
'Fortuna D\xc3\xbcsseldorf'
>>> print u'Fortuna Düsseldorf'.encode('utf8')
Fortuna Düsseldorf

The alternative is for you upgrade to Python 3; there repr() only uses escape sequences for codepoints that have no printable glyphs (control codes, reserved codepoints, surrogates, etc; if the codepoint is not a space but falls in a C* or Z* general category, it is escaped). The new ascii() function gives you the Python 2 repr() behaviour still.

More Related Contents:

  1. Why should we NOT use sys.setdefaultencoding(“utf-8”) in a py script?
  2. Why does Python print unicode characters when the default encoding is ASCII?
  3. Dangers of sys.setdefaultencoding(‘utf-8’)
  4. How do I check if a string is unicode or ascii?
  5. General Unicode/UTF-8 support for csv files in Python 2.6
  6. Why doesn’t Python recognize my utf-8 encoded source file?
  7. ElementTree and unicode
  8. Convert “\x” escaped string into readable string in python
  9. python encoding utf-8
  10. Python – dealing with mixed-encoding files
  11. SQLite, python, unicode, and non-utf data
  12. UnicodeDecodeError when performing os.walk
  13. What is the best way to remove accents (normalize) in a Python unicode string?
  14. Changing default encoding of Python?
  15. How to remove \xa0 from string in Python?
  16. Replace non-ASCII characters with a single space
  17. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xef in position 1
  18. What exactly do “u” and “r” string prefixes do, and what are raw string literals?
  19. Convert Unicode to ASCII without errors in Python
  20. Character reading from file in Python
  21. Python & MySql: Unicode and Encoding
  22. How to convert a file to utf-8 in Python?
  23. Writing UTF-8 String to MySQL with Python
  24. How to write UTF-8 in a CSV file
  25. Python 3: How to specify stdin encoding
  26. Python – Unicode to ASCII conversion
  27. Saving UTF-8 texts with json.dumps as UTF-8, not as a \u escape sequence
  28. Why does this Python program send empty emails when I encode it with utf-8? [duplicate]
  29. Changing the “locale preferred encoding” in Python 3 in Windows
  30. Printing a string prints ‘u’ before the string in Python?

Leave a Comment