How many characters can be mapped with Unicode?

by

I am asking for the count of all the possible valid combinations in Unicode with explanation.

1,111,998: 17 planes × 65,536 characters per plane – 2048 surrogates – 66 noncharacters

Note that UTF-8 and UTF-32 could theoretically encode much more than 17 planes, but the range is restricted based on the limitations of the UTF-16 encoding.

137,929 code points are actually assigned in Unicode 12.1.

I also don’t understand why continuation bytes have restrictions even though starting byte of that char clears how long it should be.

The purpose of this restriction in UTF-8 is to make the encoding self-synchronizing.

For a counterexample, consider the Chinese GB 18030 encoding. There, the letter ß is represented as the byte sequence 81 30 89 38, which contains the encoding of the digits 0 and 8. So if you have a string-searching function not designed for this encoding-specific quirk, then a search for the digit 8 will find a false positive within the letter ß.

In UTF-8, this cannot happen, because the non-overlap between lead bytes and trail bytes guarantees that the encoding of a shorter character can never occur within the encoding of a longer character.

More Related Contents:

  1. UTF-8, UTF-16, and UTF-32
  2. What’s the difference between UTF-8 and UTF-8 without BOM?
  3. What is the difference between UTF-8 and Unicode?
  4. What are Unicode, UTF-8, and UTF-16?
  5. What is the proper way to URL encode Unicode characters?
  6. FPDF utf-8 encoding (HOW-TO)
  7. Unicode, UTF, ASCII, ANSI format differences
  8. What’s the difference between Unicode and UTF-8? [duplicate]
  9. Manually converting unicode codepoints into UTF-8 and UTF-16
  10. Really Good, Bad UTF-8 example test data [closed]
  11. How does UTF-8 “variable-width encoding” work?
  12. Difference between UTF-8 and UTF-16?
  13. character showing up in files. How to remove them?
  14. What’s the difference between UTF-8 and UTF-8 with BOM?
  15. Classic ASP: How to write unicode string data in classic ASP?
  16. ASCII vs Unicode + UTF-8
  17. UTF-8 & Unicode, what’s with 0xC0 and 0x80?
  18. UTF-8 Continuation bytes
  19. Saving utf-8 texts with json.dumps as UTF8, not as \u escape sequence
  20. Why does modern Perl avoid UTF-8 by default?
  21. u’\ufeff’ in Python string
  22. Displaying Unicode in Powershell
  23. PHP: Convert unicode codepoint to UTF-8
  24. How do I check if a string is unicode or ascii?
  25. Read a UTF-8 text file with BOM
  26. C programming: How to program for Unicode?
  27. In Unicode, why are there two representations for the Arabic digits?
  28. iconv: Converting from Windows ANSI to UTF-8 with BOM
  29. python encoding utf-8
  30. Why is Unicode restricted to 0x10FFFF?

Leave a Comment