How does unicodedata.normalize(form, unistr) work?

by

I find the documentation pretty clear, but here are a few code examples:

from unicodedata import normalize

print '%r' % normalize('NFD', u'\u00C7')  # decompose: convert Ç to "C + ̧"
print '%r' % normalize('NFC', u'C\u0327') # compose: convert "C + ̧" to Ç

Both ‘D’ (=decompose) forms convert a single combined character (like ä) into two characters (a + two dots). Both ‘C’ (=compose) forms do the reverse.

The two “K” forms are used to convert characters added to Unicode for compatibility purposes. For example, to support software that cannot draw circles around symbols, there is a set of “circled numbers”, like ① (unicode number 2460). When we apply the canonical decomposition (NFD) to it, it doesn’t do anything:

print '%r' % normalize('NFD', u'\u2460')     # u'\u2460'

However, the compatibility decomposition (NFKD) will return the corresponding “compatible” character:

print '%r' % normalize('NFKD', u'\u2460')    # 1

See http://en.wikipedia.org/wiki/Unicode_equivalence for more details.

More Related Contents:

  1. Replace non-ASCII characters with a single space
  2. Windows cmd encoding change causes Python crash
  3. Why does Python print unicode characters when the default encoding is ASCII?
  4. Character reading from file in Python
  5. Get a list of all the encodings Python can encode to
  6. How do I check if a string is unicode or ascii?
  7. ‘UCS-2’ codec can’t encode characters in position 1050-1050
  8. Why doesn’t Python recognize my utf-8 encoded source file?
  9. UnicodeEncodeError: ‘ascii’ codec can’t encode character at special name [duplicate]
  10. Python 3: How to specify stdin encoding
  11. Python ascii utf unicode
  12. Python – Unicode to ASCII conversion
  13. ElementTree and unicode
  14. How to detect string byte encoding?
  15. Convert “\x” escaped string into readable string in python
  16. python encoding utf-8
  17. Python – dealing with mixed-encoding files
  18. UnicodeDecodeError when performing os.walk
  19. u’\ufeff’ in Python string
  20. Does “\d” in regex mean a digit?
  21. Writing Unicode text to a text file?
  22. How to resolve TypeError: can only concatenate str (not “int”) to str [duplicate]
  23. How to read Unicode input and compare Unicode strings in Python?
  24. UnicodeDecodeError: ‘utf8’ codec can’t decode bytes in position 3-6: invalid data
  25. Python not sorting unicode properly. Strcoll doesn’t help
  26. python-re: How do I match an alpha character
  27. Error “(unicode error) ‘unicodeescape’ codec can’t decode bytes in position 2-3: truncated \UXXXXXXXX escape” [duplicate]
  28. Matching only a unicode letter in Python re
  29. matplotlib imshow – default colour normalisation
  30. How to send non-English unicode string using HTTP header?

Leave a Comment