UTF-8 & Unicode, what’s with 0xC0 and 0x80?

by

It’s not a comparison with 0xc0, it’s a logical AND operation with 0xc0.

The bit mask 0xc0 is 11 00 00 00 so what the AND is doing is extracting only the top two bits:

    ab cd ef gh
AND 11 00 00 00
    -- -- -- --
  = ab 00 00 00

This is then compared to 0x80 (binary 10 00 00 00). In other words, the if statement is checking to see if the top two bits of the value are not equal to 10.

“Why?”, I hear you ask. Well, that’s a good question. The answer is that, in UTF-8, all bytes that begin with the bit pattern 10 are subsequent bytes of a multi-byte sequence:

                    UTF-8
Range              Encoding  Binary value
-----------------  --------  --------------------------
U+000000-U+00007f  0xxxxxxx  0xxxxxxx

U+000080-U+0007ff  110yyyxx  00000yyy xxxxxxxx
                   10xxxxxx

U+000800-U+00ffff  1110yyyy  yyyyyyyy xxxxxxxx
                   10yyyyxx
                   10xxxxxx

U+010000-U+10ffff  11110zzz  000zzzzz yyyyyyyy xxxxxxxx
                   10zzyyyy
                   10yyyyxx
                   10xxxxxx

So, what this little snippet is doing is going through every byte of your UTF-8 string and counting up all the bytes that aren’t continuation bytes (i.e., it’s getting the length of the string, as advertised). See this wikipedia link for more detail and Joel Spolsky’s excellent article for a primer.

An interesting aside by the way. You can classify bytes in a UTF-8 stream as follows:

  • With the high bit set to 0, it’s a single byte value.
  • With the two high bits set to 10, it’s a continuation byte.
  • Otherwise, it’s the first byte of a multi-byte sequence and the number of leading 1 bits indicates how many bytes there are in total for this sequence (110... means two bytes, 1110... means three bytes, etc).

More Related Contents:

  1. What’s the difference between UTF-8 and UTF-8 without BOM?
  2. What is the difference between UTF-8 and Unicode?
  3. What are Unicode, UTF-8, and UTF-16?
  4. What is the proper way to URL encode Unicode characters?
  5. UTF-8, UTF-16, and UTF-32
  6. FPDF utf-8 encoding (HOW-TO)
  7. What’s the difference between Unicode and UTF-8? [duplicate]
  8. Manually converting unicode codepoints into UTF-8 and UTF-16
  9. Really Good, Bad UTF-8 example test data [closed]
  10. How does UTF-8 “variable-width encoding” work?
  11. How many characters can be mapped with Unicode?
  12. What’s the difference between UTF-8 and UTF-8 with BOM?
  13. Classic ASP: How to write unicode string data in classic ASP?
  14. ASCII vs Unicode + UTF-8
  15. UTF-8 Continuation bytes
  16. u’\ufeff’ in Python string
  17. PHP: Convert unicode codepoint to UTF-8
  18. How do I check if a string is unicode or ascii?
  19. What are the most common non-BMP Unicode characters in actual use? [closed]
  20. strlen() and UTF-8 encoding
  21. character showing up in files. How to remove them?
  22. Python DictWriter writing UTF-8 encoded CSV files
  23. How can I get the Unicode code point(s) of a Character?
  24. python 3.0, how to make print() output unicode?
  25. Use of ‘use utf8;’ gives me ‘Wide character in print’
  26. Is there a Unicode glyph that looks like a “key” icon? [closed]
  27. Do C++11 regular expressions work with UTF-8 strings?
  28. What does u’\ufe0f’ in an emoji mean? Is it the same if I delete it?
  29. UTF8 Filenames in PHP and Different Unicode Encodings
  30. How can I use Unicode characters on the Windows command line?

Leave a Comment