What is normalized UTF-8 all about?

by

Everything You Never Wanted to Know about Unicode Normalization

Canonical Normalization

Unicode includes multiple ways to encode some characters, most notably accented characters. Canonical normalization changes the code points into a canonical encoding form. The resulting code points should appear identical to the original ones barring any bugs in the fonts or rendering engine.

When To Use

Because the results appear identical, it is always safe to apply canonical normalization to a string before storing or displaying it, as long as you can tolerate the result not being bit for bit identical to the input.

Canonical normalization comes in 2 forms: NFD and NFC. The two are equivalent in the sense that one can convert between these two forms without loss. Comparing two strings under NFC will always give the same result as comparing them under NFD.

NFD

NFD has the characters fully expanded out. This is the faster normalization form to calculate, but the results in more code points (i.e. uses more space).

If you just want to compare two strings that are not already normalized, this is the preferred normalization form unless you know you need compatibility normalization.

NFC

NFC recombines code points when possible after running the NFD algorithm. This takes a little longer, but results in shorter strings.

Compatibility Normalization

Unicode also includes many characters that really do not belong, but were used in legacy character sets. Unicode added these to allow text in those character sets to be processed as Unicode, and then be converted back without loss.

Compatibility normalization converts these to the corresponding sequence of “real” characters, and also performs canonical normalization. The results of compatibility normalization may not appear identical to the originals.

Characters that include formatting information are replaced with ones that do not. For example the character ⁹ gets converted to 9. Others don’t involve formatting differences. For example the roman numeral character Ⅸ is converted to the regular letters IX.

Obviously, once this transformation has been performed, it is no longer possible to losslessly convert back to the original character set.

When to use

The Unicode Consortium suggests thinking of compatibility normalization like a ToUpperCase transform. It is something that may be useful in some circumstances, but you should not just apply it willy-nilly.

An excellent use case would be a search engine since you would probably want a search for 9 to match ⁹.

One thing you should probably not do is display the result of applying compatibility normalization to the user.

NFKC/NFKD

Compatibility normalization form comes in two forms NFKD and NFKC. They have the same relationship as between NFD and C.

Any string in NFKC is inherently also in NFC, and the same for the NFKD and NFD. Thus NFKD(x)=NFD(NFKC(x)), and NFKC(x)=NFC(NFKD(x)), etc.

Conclusion

If in doubt, go with canonical normalization. Choose NFC or NFD based on the space/speed trade-off applicable, or based on what is required by something you are inter-operating with.

More Related Contents:

  1. Matching Unicode letter characters in PCRE/PHP
  2. Reference: Why are my “special” Unicode characters encoded weird using json_encode?
  3. Unicode character in PHP string
  4. preg_match and UTF-8 in PHP
  5. Setting the PHP default encoding to UTF-8
  6. How to get code point number for a given character in a utf-8 string?
  7. PHP: Convert unicode codepoint to UTF-8
  8. Any way to return PHP `json_encode` with encode UTF-8 and not Unicode? [duplicate]
  9. Storing and displaying unicode string (हिन्दी) using PHP and MySQL
  10. PHP decoding and encoding json with unicode characters
  11. How to get the character from unicode code point in PHP?
  12. UTF-8 to Unicode Code Points
  13. Calling C/C++ library function from PHP [closed]
  14. Multibyte trim in PHP?
  15. What factors make PHP Unicode-incompatible?
  16. Exotic names for methods, constants, variables and fields – Bug or Feature?
  17. Extending PHP with C++?
  18. Trim unicode whitespace in PHP 5.2
  19. How does similar_text work?
  20. convert Persian/Arabic numbers to English numbers
  21. strlen() and UTF-8 encoding
  22. How to open file in PHP that has unicode characters in its name?
  23. How to reverse a Unicode string
  24. Split string into sentences using regex
  25. PHP function imagettftext() and unicode
  26. How to IPC between PHP clients and a C Daemon Server?
  27. Remove or match a Unicode Zero Width Space PHP
  28. UTF8 Filenames in PHP and Different Unicode Encodings
  29. Trim unicode whitespace in PHP
  30. Is there an equivalent in C++ of PHP’s explode() function? [duplicate]

Leave a Comment