Difference between UTF-8 and UTF-16?

by

I believe there are a lot of good articles about this around the Web, but here is a short summary.

Both UTF-8 and UTF-16 are variable length encodings. However, in UTF-8 a character may occupy a minimum of 8 bits, while in UTF-16 character length starts with 16 bits.

Main UTF-8 pros:

  • Basic ASCII characters like digits, Latin characters with no accents, etc. occupy one byte which is identical to US-ASCII representation. This way all US-ASCII strings become valid UTF-8, which provides decent backwards compatibility in many cases.
  • No null bytes, which allows to use null-terminated strings, this introduces a great deal of backwards compatibility too.
  • UTF-8 is independent of byte order, so you don’t have to worry about Big Endian / Little Endian issue.

Main UTF-8 cons:

  • Many common characters have different length, which slows indexing by codepoint and calculating a codepoint count terribly.
  • Even though byte order doesn’t matter, sometimes UTF-8 still has BOM (byte order mark) which serves to notify that the text is encoded in UTF-8, and also breaks compatibility with ASCII software even if the text only contains ASCII characters. Microsoft software (like Notepad) especially likes to add BOM to UTF-8.

Main UTF-16 pros:

  • BMP (basic multilingual plane) characters, including Latin, Cyrillic, most Chinese (the PRC made support for some codepoints outside BMP mandatory), most Japanese can be represented with 2 bytes. This speeds up indexing and calculating codepoint count in case the text does not contain supplementary characters.
  • Even if the text has supplementary characters, they are still represented by pairs of 16-bit values, which means that the total length is still divisible by two and allows to use 16-bit char as the primitive component of the string.

Main UTF-16 cons:

  • Lots of null bytes in US-ASCII strings, which means no null-terminated strings and a lot of wasted memory.
  • Using it as a fixed-length encoding “mostly works” in many common scenarios (especially in US / EU / countries with Cyrillic alphabets / Israel / Arab countries / Iran and many others), often leading to broken support where it doesn’t. This means the programmers have to be aware of surrogate pairs and handle them properly in cases where it matters!
  • It’s variable length, so counting or indexing codepoints is costly, though less than UTF-8.

In general, UTF-16 is usually better for in-memory representation because BE/LE is irrelevant there (just use native order) and indexing is faster (just don’t forget to handle surrogate pairs properly). UTF-8, on the other hand, is extremely good for text files and network protocols because there is no BE/LE issue and null-termination often comes in handy, as well as ASCII-compatibility.

More Related Contents:

  1. What is a “surrogate pair” in Java?
  2. What is the Java’s internal represention for String? Modified UTF-8? UTF-16?
  3. UTF-8, UTF-16, and UTF-32
  4. Java equivalent to JavaScript’s encodeURIComponent that produces identical output?
  5. Convert escaped Unicode character back to actual character
  6. Using PDFBox to write UTF-8 encoded strings to a PDF [duplicate]
  7. Write a file in UTF-8 using FileWriter (Java)?
  8. setting a UTF-8 in java and csv file [duplicate]
  9. Why does the Java char primitive take up 2 bytes of memory?
  10. How does Java store UTF-16 characters in its 16-bit char type?
  11. Byte and char conversion in Java
  12. Java charAt used with characters that have two code units
  13. How to read UTF8 encoded file using RandomAccessFile?
  14. Byte order mark screws up file reading in Java
  15. Spring MVC UTF-8 Encoding
  16. Converting Symbols, Accent Letters to English Alphabet
  17. How to convert a string with Unicode encoding to a string of letters
  18. How can I replace non-printable Unicode characters in Java?
  19. What is the regex to extract all the emojis from a string?
  20. Valid characters in a Java class name
  21. Binary to text in Java
  22. Reading InputStream as UTF-8
  23. Java – Convert String to valid URI object
  24. Convert International String to \u Codes in java
  25. UTF-8 encoding in JSP page [duplicate]
  26. character showing up in files. How to remove them?
  27. Why can’t I use \u000D and \u000A as CR and LF in Java?
  28. How to replace � in a string
  29. How to correctly write UTF-8 strings into MySQL through JDBC interface
  30. Java: Convert String “\uFFFF” into char

Leave a Comment