How does UTF-8 “variable-width encoding” work?

by

Each byte starts with a few bits that tell you whether it’s a single byte code-point, a multi-byte code point, or a continuation of a multi-byte code point. Like this:

0xxx xxxx    A single-byte US-ASCII code (from the first 127 characters)

The multi-byte code-points each start with a few bits that essentially say “hey, you need to also read the next byte (or two, or three) to figure out what I am.” They are:

110x xxxx    One more byte follows
1110 xxxx    Two more bytes follow
1111 0xxx    Three more bytes follow

Finally, the bytes that follow those start codes all look like this:

10xx xxxx    A continuation of one of the multi-byte characters

Since you can tell what kind of byte you’re looking at from the first few bits, then even if something gets mangled somewhere, you don’t lose the whole sequence.

More Related Contents:

  1. What’s the difference between UTF-8 and UTF-8 without BOM?
  2. What is the difference between UTF-8 and Unicode?
  3. What is the proper way to URL encode Unicode characters?
  4. FPDF utf-8 encoding (HOW-TO)
  5. What’s the difference between UTF-8 and UTF-8 with BOM?
  6. ASCII vs Unicode + UTF-8
  7. Trouble with UTF-8 characters; what I see is not what I stored
  8. UTF8 to/from wide char conversion in STL
  9. What are Unicode, UTF-8, and UTF-16?
  10. Convert Unicode to ASCII without errors in Python
  11. UTF-8, UTF-16, and UTF-32
  12. Unicode, UTF, ASCII, ANSI format differences
  13. Printing UTF-8 strings with printf – wide vs. multibyte string literals
  14. What’s the difference between Unicode and UTF-8? [duplicate]
  15. Manually converting unicode codepoints into UTF-8 and UTF-16
  16. Really Good, Bad UTF-8 example test data [closed]
  17. PowerShell script runs when pasted into the PowerShell window, but not when run from shortcut
  18. How many characters can be mapped with Unicode?
  19. Read a UTF-8 text file with BOM
  20. C programming: How to program for Unicode?
  21. What is the range of Unicode Printable Characters?
  22. Ruby 1.9: how can I properly upcase & downcase multibyte strings?
  23. Classic ASP: How to write unicode string data in classic ASP?
  24. UTF-8 & Unicode, what’s with 0xC0 and 0x80?
  25. UTF-8 Continuation bytes
  26. Inheriting and overriding functions of a std::string?
  27. Bytes in a unicode Python string
  28. Why is Unicode restricted to 0x10FFFF?
  29. Why does modern Perl avoid UTF-8 by default?
  30. Decoding numeric html entities via PHP

Leave a Comment