How is unicode represented internally in Python?

by

I’m assuming you want to know about CPython, the standard implementation. Python 2 and Python 3.0-3.2 use either UCS2* or UCS4 for Unicode characters, meaning it’ll either use 2 bytes or 4 bytes for each character. Which one is picked is a compile-time option.

\u2049 is then represented as either \x49\x20 or \x20\x49 or \x49\x20\x00\x00 or \x00\x00\x20\x49 depending on the native byte order of your system and if UCS2 or UCS4 was picked. ASCII characters in a unicode string still use 2 or 4 bytes per character too.

Python 3.3 switched to a new internal representation, using the most compact form needed to represent all characters in a string. Either 1 byte, 2 bytes or 4 bytes are picked. ASCII and Latin-1 text uses just 1 byte per character, the rest of the BMP characters require 2 bytes and after that 4 bytes is used.

See PEP-393: Flexible String Representation for the full low-down on these representations.

* Technically speaking the UCS-2 build uses UTF-16, as non-BMP characters use UTF-16 surrogates to encode to 4 bytes (2 UTF-16 characters) each. However, Python documentation still refers to this as UCS2.

This does lead to unexpected behaviour such as the len() on non-BMP unicode strings being longer than the number of characters contained.

More Related Contents:

  1. What does the ‘b’ character do in front of a string literal?
  2. Python string interning
  3. Convert a Unicode string to a string in Python (containing extra symbols)
  4. About the changing id of an immutable string
  5. removing emojis from a string in Python
  6. How to check if a string in Python is in ASCII?
  7. Suppress the u’prefix indicating unicode’ in python strings
  8. Python str vs unicode types
  9. Python __str__ versus __unicode__
  10. What is the difference between encode/decode?
  11. Python string with space and without space at the end and immutability
  12. Why is True returned when checking if an empty string is in another?
  13. byte string vs. unicode string. Python
  14. Python string literal concatenation
  15. Why is variable1 += variable2 much faster than variable1 = variable1 + variable2?
  16. Identifier normalization: Why is the micro sign converted into the Greek letter mu?
  17. What does ‘u’ mean in a list?
  18. How do I get str.translate to work with Unicode strings?
  19. How to detect string byte encoding?
  20. Python string to unicode [duplicate]
  21. What is internal representation of string in Python 3.x
  22. How does str(list) work?
  23. Remove unwanted parts from strings in a column
  24. Good Python modules for fuzzy string comparison? [closed]
  25. Regex for existence of some words whose order doesn’t matter
  26. How to make Django slugify work properly with Unicode strings?
  27. How to explain the str.maketrans function in Python 3.6?
  28. How to remove symbols from a string with Python? [duplicate]
  29. How to check a string for a special character?
  30. Why are str.count(”) and len(str) giving different outputs when used on an empty string?

Leave a Comment