Python – Unicode to ASCII conversion

by

The Unicode characters u'\xce0' and u'\xc9' do not have any corresponding ASCII values. So, if you don’t want to lose data, you have to encode that data in some way that’s valid as ASCII. Options include:

>>> print s.encode('ascii', errors="backslashreplace")
ABRA\xc3O JOS\xc9
>>> print s.encode('ascii', errors="xmlcharrefreplace")
ABRAÃO JOSÉ
>>> print s.encode('unicode-escape')
ABRA\xc3O JOS\xc9
>>> print s.encode('punycode')
ABRAO JOS-jta5e

All of these are ASCII strings, and contain all of the information from your original Unicode string (so they can all be reversed without loss of data), but none of them are all that pretty for an end-user (and none of them can be reversed just by decode('ascii')).

See str.encode, Python Specific Encodings, and Unicode HOWTO for more info.

As a side note, when some people say “ASCII”, they really don’t mean “ASCII” but rather “any 8-bit character set that’s a superset of ASCII” or “some particular 8-bit character set that I have in mind”. If that’s what you meant, the solution is to encode to the right 8-bit character set:

>>> s.encode('utf-8')
'ABRA\xc3\x83O JOS\xc3\x89'
>>> s.encode('cp1252')
'ABRA\xc3O JOS\xc9'
>>> s.encode('iso-8859-15')
'ABRA\xc3O JOS\xc9'

The hard part is knowing which character set you meant. If you’re writing both the code that produces the 8-bit strings and the code that consumes it, and you don’t know any better, you meant UTF-8. If the code that consumes the 8-bit strings is, say, the open function or a web browser that you’re serving a page to or something else, things are more complicated, and there’s no easy answer without a lot more information.

More Related Contents:

  1. Replace non-ASCII characters with a single space
  2. Why does Python print unicode characters when the default encoding is ASCII?
  3. Character reading from file in Python
  4. UnicodeEncodeError: ‘ascii’ codec can’t encode character at special name [duplicate]
  5. How can I remove non-ASCII characters but leave periods and spaces?
  6. How to check if a string in Python is in ASCII?
  7. Convert Unicode to ASCII without errors in Python
  8. Python string prints as [u’String’]
  9. Windows cmd encoding change causes Python crash
  10. Python UnicodeDecodeError – Am I misunderstanding encode?
  11. Python Unicode Encode Error
  12. Get a list of all the encodings Python can encode to
  13. Where is Python’s “best ASCII for this Unicode” database? [closed]
  14. Non-ASCII characters in Matplotlib
  15. How do I check if a string is unicode or ascii?
  16. How to print UTF-8 encoded text to the console in Python < 3?
  17. ‘UCS-2’ codec can’t encode characters in position 1050-1050
  18. Why doesn’t Python recognize my utf-8 encoded source file?
  19. In Python, how do I decode GZIP encoding?
  20. Python 3: How to specify stdin encoding
  21. Python ascii utf unicode
  22. How do I treat an ASCII string as unicode and unescape the escaped characters in it in python?
  23. ElementTree and unicode
  24. How to detect string byte encoding?
  25. Convert “\x” escaped string into readable string in python
  26. python encoding utf-8
  27. How does unicodedata.normalize(form, unistr) work?
  28. Python – dealing with mixed-encoding files
  29. UnicodeDecodeError when performing os.walk
  30. Python 3 chokes on CP-1252/ANSI reading

Leave a Comment