UTF-8 Continuation bytes

by

A continuation byte in UTF-8 is any byte where the top two bits are 10.

They are the subsequent bytes in multi-byte sequences. The following table may help:

Unicode code points  Encoding  Binary value
-------------------  --------  ------------
 U+000000-U+00007f   0xxxxxxx  0xxxxxxx

 U+000080-U+0007ff   110yyyxx  00000yyy xxxxxxxx
                     10xxxxxx

 U+000800-U+00ffff   1110yyyy  yyyyyyyy xxxxxxxx
                     10yyyyxx
                     10xxxxxx

 U+010000-U+10ffff   11110zzz  000zzzzz yyyyyyyy xxxxxxxx
                     10zzyyyy
                     10yyyyxx
                     10xxxxxx

Here you can see how the Unicode code points map to UTF-8 multi-byte byte sequences, and their equivalent binary values.

The basic rules are this:

  1. If a byte starts with a 0 bit, it’s a single byte value less than 128.
  2. If it starts with 11, it’s the first byte of a multi-byte sequence and the number of 1 bits at the start indicates how many bytes there are in total (110xxxxx has two bytes, 1110xxxx has three and 11110xxx has four).
  3. If it starts with 10, it’s a continuation byte.

This distinction allows quite handy processing such as being able to back up from any byte in a sequence to find the first byte of that code point. Just search backwards until you find one not beginning with the 10 bits.

Similarly, it can also be used for a UTF-8 strlen by only counting non-10xxxxxx bytes.

More Related Contents:

  1. What’s the difference between UTF-8 and UTF-8 without BOM?
  2. What is the difference between UTF-8 and Unicode?
  3. What are Unicode, UTF-8, and UTF-16?
  4. What is the proper way to URL encode Unicode characters?
  5. UTF-8, UTF-16, and UTF-32
  6. FPDF utf-8 encoding (HOW-TO)
  7. What’s the difference between Unicode and UTF-8? [duplicate]
  8. Manually converting unicode codepoints into UTF-8 and UTF-16
  9. Really Good, Bad UTF-8 example test data [closed]
  10. How does UTF-8 “variable-width encoding” work?
  11. How many characters can be mapped with Unicode?
  12. What’s the difference between UTF-8 and UTF-8 with BOM?
  13. Classic ASP: How to write unicode string data in classic ASP?
  14. ASCII vs Unicode + UTF-8
  15. UTF-8 & Unicode, what’s with 0xC0 and 0x80?
  16. Unicode (UTF-8) reading and writing to files in Python
  17. What is the Windows equivalent for en_US.UTF-8 locale?
  18. Is there a Windows command shell that will display Unicode characters?
  19. Python and BeautifulSoup encoding issues [duplicate]
  20. How do I replace accented Latin characters in Ruby?
  21. MySQL “incorrect string value” error when save unicode string in Django
  22. Light C Unicode Library [closed]
  23. UTF-8 to Unicode Code Points
  24. What’s the best option to display Unicode text (hebrew, etc.) in VB6
  25. Unicode range for Japanese
  26. MySQL C# Text Encoding Problems
  27. Printing Unicode characters to the PowerShell prompt
  28. Difference between Big Endian and little Endian Byte order
  29. What does u’\ufe0f’ in an emoji mean? Is it the same if I delete it?
  30. Bytes in a unicode Python string

Leave a Comment