Remove punctuation from Unicode formatted strings

by

You could use unicode.translate() method:

import unicodedata
import sys

tbl = dict.fromkeys(i for i in xrange(sys.maxunicode)
                      if unicodedata.category(unichr(i)).startswith('P'))
def remove_punctuation(text):
    return text.translate(tbl)

You could also use r'\p{P}' that is supported by regex module:

import regex as re

def remove_punctuation(text):
    return re.sub(ur"\p{P}+", "", text)

More Related Contents:

  1. What is the best way to remove accents (normalize) in a Python unicode string?
  2. How to set sys.stdout encoding in Python 3?
  3. Replace non-ASCII characters with a single space
  4. Python regex matching Unicode properties
  5. Convert Unicode to ASCII without errors in Python
  6. How to correct TypeError: Unicode-objects must be encoded before hashing?
  7. Python UnicodeDecodeError – Am I misunderstanding encode?
  8. Reading Unicode file data with BOM chars in Python
  9. UnicodeDecodeError, invalid continuation byte
  10. How to make the python interpreter correctly handle non-ASCII characters in string operations?
  11. Get a list of all the encodings Python can encode to
  12. Writing UTF-8 String to MySQL with Python
  13. urllib2 read to Unicode
  14. ‘UCS-2’ codec can’t encode characters in position 1050-1050
  15. Why doesn’t Python recognize my utf-8 encoded source file?
  16. remove unicode emoji using re in python
  17. How do I read an image from a path with Unicode characters?
  18. How to make Django slugify work properly with Unicode strings?
  19. Python ascii utf unicode
  20. latin-1 to ascii
  21. Can’t open Unicode URL with Python
  22. python: unicode in Windows terminal, encoding used?
  23. Python 3 UnicodeDecodeError: ‘charmap’ codec can’t decode byte 0x9d
  24. How do I get str.translate to work with Unicode strings?
  25. How to encode Python 3 string using \u escape code?
  26. Python returns length of 2 for single Unicode character string
  27. SQLite, python, unicode, and non-utf data
  28. UnicodeDecodeError when performing os.walk
  29. Remove non-ASCII characters from a string using python / django
  30. Python Unicode string stored as ‘\u84b8\u6c7d\u5730’ in file, how to convert it back to Unicode?

Leave a Comment