Removing control characters from a string in python

by

There are hundreds of control characters in unicode. If you are sanitizing data from the web or some other source that might contain non-ascii characters, you will need Python’s unicodedata module. The unicodedata.category(…) function returns the unicode category code (e.g., control character, whitespace, letter, etc.) of any character. For control characters, the category always starts with “C”.

This snippet removes all control characters from a string.

import unicodedata
def remove_control_characters(s):
    return "".join(ch for ch in s if unicodedata.category(ch)[0]!="C")

Examples of unicode categories:

>>> from unicodedata import category
>>> category('\r')      # carriage return --> Cc : control character
'Cc'
>>> category('\0')      # null character ---> Cc : control character
'Cc'
>>> category('\t')      # tab --------------> Cc : control character
'Cc'
>>> category(' ')       # space ------------> Zs : separator, space
'Zs'
>>> category(u'\u200A') # hair space -------> Zs : separator, space
'Zs'
>>> category(u'\u200b') # zero width space -> Cf : control character, formatting
'Cf'
>>> category('A')       # letter "A" -------> Lu : letter, uppercase
'Lu'
>>> category(u'\u4e21') # 両 ---------------> Lo : letter, other
'Lo'
>>> category(',')       # comma  -----------> Po : punctuation
'Po'
>>>

More Related Contents:

  1. Unable to use convert strings into integers for a for loop
  2. User input integer list [duplicate]
  3. Convert bytes to a string
  4. Best way to convert string to bytes in Python 3?
  5. How to search and replace text in a file?
  6. TypeError: a bytes-like object is required, not ‘str’ when writing to a file in Python 3
  7. How to use string.replace() in python 3.x
  8. TypeError: not all arguments converted during string formatting python
  9. String formatting in Python 3
  10. Find column whose name contains a specific string
  11. Pandas Extract Number from String
  12. How do I format a string using a dictionary in python-3.x?
  13. What’s the difference between str.isdigit, isnumeric and isdecimal in python?
  14. Why are str.count(”) and len(str) giving different output?
  15. How to match any string from a list of strings in regular expressions in python?
  16. How do I change the string representation of a Python class? [duplicate]
  17. Safely evaluate simple string equation
  18. Python 3.1.1 string to hex
  19. Count number of words per row
  20. How to print like printf in Python3?
  21. What’s the difference between str.isdigit(), isnumeric() and isdecimal() in Python?
  22. “TypeError: a bytes-like object is required, not ‘str'” when handling file content in Python 3
  23. How can I use f-string with a variable, not with a string literal?
  24. String-based enum in Python
  25. FutureWarning: The default value of regex will change from True to False in a future version
  26. How to sort a list of strings in reverse order without using reverse=True parameter?
  27. What is internal representation of string in Python 3.x
  28. Finding the position of a word in a string
  29. Why are str.count(”) and len(str) giving different outputs when used on an empty string?
  30. Convert bytes to a string in python 3

Leave a Comment