Why is Unicode restricted to 0x10FFFF?

by

It’s because of UTF-16. Characters outside of the base multilingual plane (BMP) are represented using a surrogate pair in UTF-16 with the first code unit (CU) lies between 0xD800–0xDBFF and the second one between 0xDC00–0xDFFF. Each of the CU represents 10 bits of the code point, allowing total 20 bits of data (0x100000 characters) which is split into 16 planes (16×216 characters). The remaining BMP will represent 0x10000 characters (code points 0–0xFFFF)

Therefore the total number of characters is 17×216 = 0x100000 + 0x10000 = 0x110000 which allows for code points from 0 to 0x110000 – 1 = 0x10FFFF. Alternatively the last representable code point can be calculated like this: Code points in the BMP are in the range 0–0xFFFF, so the offset for characters encoded with a surrogate pair is 0xFFFF + 1 = 0x10000, which means the last code point that a surrogate pair represents is 0xFFFFF + 0x10000 = 0x10FFFF

That’s guaranteed by Unicode Character Encoding Stability Policies that a code point above that will never be assigned

The General_Category property value Surrogate (Cs) is immutable: the set of code points with that value will never change.

Historically UTF-8 allows up to U+7FFFFFFF using 6 bytes whereas UTF-32 can store twice the number of that. However due to the limit in UTF-16 the Unicode committee has decided that UTF-8 can never be longer than 4 bytes, resulting in the same range as UTF-16

In November 2003, UTF-8 was restricted by RFC 3629 to match the constraints of the UTF-16 character encoding: explicitly prohibiting code points corresponding to the high and low surrogate characters removed more than 3% of the three-byte sequences, and ending at U+10FFFF removed more than 48% of the four-byte sequences and all five- and six-byte sequences.

https://en.wikipedia.org/wiki/UTF-8#History

The same has been applied to UTF-32

In November 2003, Unicode was restricted by RFC 3629 to match the constraints of the UTF-16 encoding: explicitly prohibiting code points greater than U+10FFFF (and also the high and low surrogates U+D800 through U+DFFF). This limited subset defines UTF-32

https://en.wikipedia.org/wiki/UTF-32

You can read this more detailed answer and

More Related Contents:

  1. What’s the difference between UTF-8 and UTF-8 without BOM?
  2. What is the difference between UTF-8 and Unicode?
  3. What is the proper way to URL encode Unicode characters?
  4. FPDF utf-8 encoding (HOW-TO)
  5. Unicode, UTF, ASCII, ANSI format differences
  6. How does UTF-8 “variable-width encoding” work?
  7. What are the most common non-BMP Unicode characters in actual use? [closed]
  8. What’s the difference between UTF-8 and UTF-8 with BOM?
  9. What is the range of Unicode Printable Characters?
  10. ASCII vs Unicode + UTF-8
  11. Is there a Windows command shell that will display Unicode characters?
  12. PHP decoding and encoding json with unicode characters
  13. Get unicode code point of a character using Python
  14. How can I clean source code files of invisible characters?
  15. Can UTF-8 contain zero byte?
  16. List of all unicode’s open/close brackets?
  17. grepping binary files and UTF16
  18. How to determine encoding table of a text file
  19. Printing Unicode characters to the PowerShell prompt
  20. Difference between Big Endian and little Endian Byte order
  21. UTF-8 text (Hindi) not getting displayed on Browser window or Eclipse console
  22. How to read utf16 text file to string in golang?
  23. What does it mean when my text is displayed as Question Marks?
  24. UTF-8 & Unicode, what’s with 0xC0 and 0x80?
  25. Specification of source charset encoding in MSVC++, like gcc “-finput-charset=CharSet”
  26. MySQL: Get character-set of database or table or column?
  27. Issue with smtplib sending mail with unicode characters in Python 3.1
  28. Input unicode string with pyautogui
  29. unicode().decode(‘utf-8’, ‘ignore’) raising UnicodeEncodeError
  30. How to correctly compute the length of a String in Java?

Leave a Comment