It’s because of UTF-16. Characters outside of the base multilingual plane (BMP) are represented using a surrogate pair in UTF-16 with the first code unit (CU) lies between 0xD800–0xDBFF and the second one between 0xDC00–0xDFFF. Each of the CU represents 10 bits of the code point, allowing total 20 bits of data (0x100000 characters) which … Read more
Provided that all the characters that you’re catering for exist in the Basic Multilingual Plane (it’s unlikely that you’ll need anything more), then a simple UTF-16 encoding should suffice. Wikipedia: All possible code points from U+0000 through U+10FFFF, except for the surrogate code points U+D800–U+DFFF (which are not characters), are uniquely mapped by UTF-16 regardless … Read more
If I understand your question correctly, you can do this. >>> s=”㈲” >>> s.encode(“unicode_escape”) b’\\u3232′ Shows the unicode escape code as a source string.
Emoji are now the most common non-BMP characters by far. 😂, otherwise known as U+1F602 FACE WITH TEARS OF JOY, is the most common one on Twitter’s public stream. It occurs more frequently than the tilde!