Python returns length of 2 for single Unicode character string

by

Your Python binary was compiled with UCS-2 support (a narrow build) and internally anything outside of the BMP (Basic Multilingual Plane) is represented using a surrogate pair.

That means such codepoints show up as 2 characters when asking for the length.

You’ll have to recompile your Python binary to use UCS-4 instead if this matters (./configure --enable-unicode=ucs4 will enable it), or upgrade to Python 3.3 or newer, where Python’s Unicode support was overhauled to use a variable-width Unicode type that switches between ASCII, UCS-2 and UCS-4 as required by the codepoints contained.

On Python versions 2.7 and 3.0 – 3.2, you can detect what kind of build you have by inspecting the sys.maxunicode value; it’ll be 2^16-1 == 65535 == 0xFFFF for a narrow UCS-2 build, 1114111 == 0x10FFFF for a wide UCS-4 build. In Python 3.3 and up it is always set to 1114111.

Demo:

# Narrow build
$ bin/python -c 'import sys; print sys.maxunicode, len(u"\U0001f44d"), list(u"\U0001f44d")'
65535 2 [u'\ud83d', u'\udc4d']
# Wide build
$ python -c 'import sys; print sys.maxunicode, len(u"\U0001f44d"), list(u"\U0001f44d")'
1114111 1 [u'\U0001f44d']

More Related Contents:

  1. Python – ‘ascii’ codec can’t decode byte
  2. Why does ENcoding a string result in a DEcoding error (UnicodeDecodeError)?
  3. UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\xa0′ in position 20: ordinal not in range(128)
  4. How to remove \xa0 from string in Python?
  5. SyntaxError: Non-ASCII character ‘\xa3’ in file when function returns ‘£’
  6. Read and Write CSV files including unicode with Python 2.7
  7. Removing unicode \u2026 like characters in a string in python2.7 [duplicate]
  8. How to convert a string to utf-8 in Python
  9. How to read Unicode input and compare Unicode strings in Python?
  10. Get unicode code point of a character using Python
  11. “SyntaxError: Non-ASCII character …” or “SyntaxError: Non-UTF-8 code starting with …” trying to use non-ASCII text in a Python script
  12. How can I use io.StringIO() with the csv module?
  13. Python string to unicode [duplicate]
  14. How to sort a dataFrame in python pandas by two or more columns?
  15. Why isn’t my Pandas ‘apply’ function referencing multiple columns working? [closed]
  16. Why does Python print unicode characters when the default encoding is ASCII?
  17. Split a python list into other “sublists” i.e smaller lists [duplicate]
  18. What is the right way to debug in iPython notebook?
  19. How to get current time in python and break up into year, month, day, hour, minute?
  20. How can I open two consoles from a single script
  21. Python 3: How to specify stdin encoding
  22. Python 2.7: Print to File
  23. Python to automatically select serial ports (for Arduino)
  24. How do I treat an ASCII string as unicode and unescape the escaped characters in it in python?
  25. Python unicode regular expression matching failing with some unicode characters -bug or mistake?
  26. Why does updating one dictionary object affect other?
  27. how to open chrome in incognito mode from Python
  28. Matplotlib just giving error messages
  29. ImportError: DLL load failed when importing Numpy installed in conda virtual environment
  30. How to handle IncompleteRead: in python

Leave a Comment