Bytes in a unicode Python string

by

In Python 2, Unicode strings may contain both unicode and bytes:

No, they may not. They contain Unicode characters.

Within the original string, \xd0 is not a byte that’s part of a UTF-8 encoding. It is the Unicode character with code point 208. u'\xd0' == u'\u00d0'. It just happens that the repr for Unicode strings in Python 2 prefers to represent characters with \x escapes where possible (i.e. code points < 256).

There is no way to look at the string and tell that the \xd0 byte is supposed to be part of some UTF-8 encoded character, or if it actually stands for that Unicode character by itself.

However, if you assume that you can always interpret those values as encoded ones, you could try writing something that analyzes each character in turn (use ord to convert to a code-point integer), decodes characters < 256 as UTF-8, and passes characters >= 256 as they were.

More Related Contents:

  1. Convert Unicode to ASCII without errors in Python
  2. Working with UTF-8 encoding in Python source [duplicate]
  3. Reading a UTF8 CSV file with Python
  4. How to remove \xa0 from string in Python?
  5. Unicode (UTF-8) reading and writing to files in Python
  6. Writing Unicode text to a text file?
  7. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xef in position 1
  8. Python & MySql: Unicode and Encoding
  9. Python and BeautifulSoup encoding issues [duplicate]
  10. Write to UTF-8 file in Python
  11. Get a list of all the encodings Python can encode to
  12. How to convert a string to utf-8 in Python
  13. Writing UTF-8 String to MySQL with Python
  14. What is the difference between encode/decode?
  15. General Unicode/UTF-8 support for csv files in Python 2.6
  16. MySQL “incorrect string value” error when save unicode string in Django
  17. Why doesn’t Python recognize my utf-8 encoded source file?
  18. How to correctly parse UTF-8 encoded HTML to Unicode strings with BeautifulSoup? [duplicate]
  19. How to unquote a urlencoded unicode string in python?
  20. Python 3 print() function with Farsi/Arabic characters [duplicate]
  21. Python ascii utf unicode
  22. Python DictWriter writing UTF-8 encoded CSV files
  23. ElementTree and unicode
  24. Saving UTF-8 texts with json.dumps as UTF-8, not as a \u escape sequence
  25. python 3 open() default encoding
  26. Convert UTF-16 to UTF-8 and remove BOM?
  27. Convert “\x” escaped string into readable string in python
  28. Python – dealing with mixed-encoding files
  29. SQLite, python, unicode, and non-utf data
  30. UnicodeDecodeError when performing os.walk

Leave a Comment