How to make the python interpreter correctly handle non-ASCII characters in string operations?

by

Throw out all characters that can’t be interpreted as ASCII:

def remove_non_ascii(s):
    return "".join(c for c in s if ord(c)<128)

Keep in mind that this is guaranteed to work with the UTF-8 encoding (because all bytes in multi-byte characters have the highest bit set to 1).

More Related Contents:

  1. What exactly do “u” and “r” string flags do, and what are raw string literals?
  2. What does the ‘b’ character do in front of a string literal?
  3. How to remove \xa0 from string in Python?
  4. How to get string objects instead of Unicode from JSON?
  5. Python string prints as [u’String’]
  6. Why does Python print unicode characters when the default encoding is ASCII?
  7. Python str vs unicode types
  8. Non-ASCII characters in Matplotlib
  9. How do I check if a string is unicode or ascii?
  10. Conversion of strings like \\uXXXX in python
  11. How to read Unicode input and compare Unicode strings in Python?
  12. Python 3 print() function with Farsi/Arabic characters [duplicate]
  13. Python – Unicode to ASCII conversion
  14. Python DictWriter writing UTF-8 encoded CSV files
  15. Python not sorting unicode properly. Strcoll doesn’t help
  16. python-re: How do I match an alpha character
  17. What does ‘u’ mean in a list?
  18. Saving UTF-8 texts with json.dumps as UTF-8, not as a \u escape sequence
  19. Error “(unicode error) ‘unicodeescape’ codec can’t decode bytes in position 2-3: truncated \UXXXXXXXX escape” [duplicate]
  20. How is unicode represented internally in Python?
  21. Unicode identifiers in Python?
  22. Printing a string prints ‘u’ before the string in Python?
  23. Matching only a unicode letter in Python re
  24. How does unicodedata.normalize(form, unistr) work?
  25. Python – dealing with mixed-encoding files
  26. Python: How can I replace full-width characters with half-width characters?
  27. Python the same char not equals
  28. SQLite, python, unicode, and non-utf data
  29. Remove non-ASCII characters from a string using python / django
  30. Python Unicode string stored as ‘\u84b8\u6c7d\u5730’ in file, how to convert it back to Unicode?

Leave a Comment