Stripping non printable characters from a string in python

by

Iterating over strings is unfortunately rather slow in Python. Regular expressions are over an order of magnitude faster for this kind of thing. You just have to build the character class yourself. The unicodedata module is quite helpful for this, especially the unicodedata.category() function. See Unicode Character Database for descriptions of the categories.

import unicodedata, re, itertools, sys

all_chars = (chr(i) for i in range(sys.maxunicode))
categories = {'Cc'}
control_chars="".join(c for c in all_chars if unicodedata.category(c) in categories)
# or equivalently and much more efficiently
control_chars="".join(map(chr, itertools.chain(range(0x00,0x20), range(0x7f,0xa0))))

control_char_re = re.compile('[%s]' % re.escape(control_chars))

def remove_control_chars(s):
    return control_char_re.sub('', s)

For Python2

import unicodedata, re, sys

all_chars = (unichr(i) for i in xrange(sys.maxunicode))
categories = {'Cc'}
control_chars="".join(c for c in all_chars if unicodedata.category(c) in categories)
# or equivalently and much more efficiently
control_chars="".join(map(unichr, range(0x00,0x20) + range(0x7f,0xa0)))

control_char_re = re.compile('[%s]' % re.escape(control_chars))

def remove_control_chars(s):
    return control_char_re.sub('', s)

For some use-cases, additional categories (e.g. all from the control group might be preferable, although this might slow down the processing time and increase memory usage significantly. Number of characters per category:

  • Cc (control): 65
  • Cf (format): 161
  • Cs (surrogate): 2048
  • Co (private-use): 137468
  • Cn (unassigned): 836601

Edit Adding suggestions from the comments.

More Related Contents:

  1. Pythonic way to create a long multi-line string
  2. How to get the value of a variable given its name in a string? [duplicate]
  3. About the changing id of an immutable string
  4. String replace doesn’t appear to be working
  5. How to delete a character from a string using Python
  6. Remove all special characters, punctuation and spaces from string
  7. Check if a word is in a string in Python
  8. Find column whose name contains a specific string
  9. How to check a string for specific characters?
  10. Proper indentation for multiline strings?
  11. Why are str.count(”) and len(str) giving different output?
  12. How do I pad a string with zeroes?
  13. How to check if a string contains an element from a list in Python
  14. How can I escape latex code received through user input?
  15. Confused about __str__ on list in Python [duplicate]
  16. write() versus writelines() and concatenated strings
  17. Python 3.1.1 string to hex
  18. Python: Using vars() to assign a string to a variable
  19. removing newlines from messy strings in pandas dataframe cells?
  20. String exact match
  21. How to strip a specific word from a string?
  22. How to detect lowercase letters in Python?
  23. Python- how do I use re to match a whole string [duplicate]
  24. How to apply __str__ function when printing a list of objects in Python [duplicate]
  25. Plural String Formatting
  26. Search for string allowing for one mismatch in any location of the string
  27. Setting the fmt option in numpy.savetxt
  28. Validating user input strings in Python
  29. How does str(list) work?
  30. How to remove all integer values from a list in python

Leave a Comment