Can UTF-8 contain zero byte?

by

Yes, the zero byte in UTF8 is code point 0, NUL. There is no other Unicode code point that will be encoded in UTF8 with a zero byte anywhere within it.

The possible code points and their UTF8 encoding are:

Range              Encoding  Binary value
-----------------  --------  --------------------------
U+000000-U+00007f  0xxxxxxx  0xxxxxxx

U+000080-U+0007ff  110yyyxx  00000yyy xxxxxxxx
                   10xxxxxx

U+000800-U+00ffff  1110yyyy  yyyyyyyy xxxxxxxx
                   10yyyyxx
                   10xxxxxx

U+010000-U+10ffff  11110zzz  000zzzzz yyyyyyyy xxxxxxxx
                   10zzyyyy
                   10yyyyxx
                   10xxxxxx

You can see that all the non-zero ASCII characters are represented as themselves while all mutibyte sequences have a high bit of 1 in all their bytes.

You may need to be careful that your ascii plaintext protocol doesn’t treat non-ASCII characters badly (since that will be all non-ASCII code points).

More Related Contents:

  1. What’s the complete range for Chinese characters in Unicode?
  2. What is the difference between UTF-8 and Unicode?
  3. What are Unicode, UTF-8, and UTF-16?
  4. Using awk to remove the Byte-order mark
  5. What is the proper way to URL encode Unicode characters?
  6. UTF-8, UTF-16, and UTF-32
  7. FPDF utf-8 encoding (HOW-TO)
  8. What’s the difference between Unicode and UTF-8? [duplicate]
  9. Manually converting unicode codepoints into UTF-8 and UTF-16
  10. Really Good, Bad UTF-8 example test data [closed]
  11. How does UTF-8 “variable-width encoding” work?
  12. How many characters can be mapped with Unicode?
  13. What are the most common non-BMP Unicode characters in actual use? [closed]
  14. Should I use accented characters in URLs?
  15. How can I get the Unicode code point(s) of a Character?
  16. python 3.0, how to make print() output unicode?
  17. What is the range of Unicode Printable Characters?
  18. iconv: Converting from Windows ANSI to UTF-8 with BOM
  19. complete, monospaced Unicode font? [closed]
  20. Is there a Unicode glyph that looks like a “key” icon? [closed]
  21. Convert unicode code point to literal character in Go
  22. Using box-drawing Unicode characters in batch files
  23. ASCII vs Unicode + UTF-8
  24. UTF-8 & Unicode, what’s with 0xC0 and 0x80?
  25. What does u’\ufe0f’ in an emoji mean? Is it the same if I delete it?
  26. Unicode output on Windows command line?
  27. How can I use Unicode characters on the Windows command line?
  28. Issue with smtplib sending mail with unicode characters in Python 3.1
  29. Input unicode string with pyautogui
  30. unicode().decode(‘utf-8’, ‘ignore’) raising UnicodeEncodeError

Leave a Comment