Splitting a string into words and punctuation

by

This is more or less the way to do it:

>>> import re
>>> re.findall(r"[\w']+|[.,!?;]", "Hello, I'm a string!")
['Hello', ',', "I'm", 'a', 'string', '!']

The trick is, not to think about where to split the string, but what to include in the tokens.

Caveats:

  • The underscore (_) is considered an inner-word character. Replace \w, if you don’t want that.
  • This will not work with (single) quotes in the string.
  • Put any additional punctuation marks you want to use in the right half of the regular expression.
  • Anything not explicitely mentioned in the re is silently dropped.

More Related Contents:

  1. Split Strings into words with multiple word boundary delimiters
  2. Split string every nth character?
  3. Split string with multiple delimiters in Python [duplicate]
  4. Split a string by a delimiter in python
  5. Split string on whitespace in Python [duplicate]
  6. How to split a string of space separated numbers into integers?
  7. Why are empty strings returned in split() results?
  8. How do I split a multi-line string into multiple lines?
  9. Reading a text file and splitting it into single words in python
  10. Product code looks like abcd2343, how to split by letters and numbers?
  11. Get last “column” after .str.split() operation on column in pandas DataFrame
  12. Splitting a semicolon-separated string to a dictionary, in Python
  13. Splitting on last delimiter in Python string?
  14. Python split string into multiple string [duplicate]
  15. When splitting an empty string in Python, why does split() return an empty list while split(‘\n’) returns [”]?
  16. Splitting a string by list of indices
  17. Split a string only by first space in python
  18. How to split strings inside a list by whitespace characters
  19. Convert string to list. Python [string.split() acting weird]
  20. How would I get everything before a : in a string Python
  21. Python regex: splitting on pattern match that is an empty string
  22. Convert list of strings to dictionary
  23. Is there a way to split a string by every nth separator in Python?
  24. How can I split a string of a mathematical expressions in python?
  25. How to split a Python string on new line characters [duplicate]
  26. Split string at nth occurrence of a given character
  27. Split a string into 2 in Python
  28. How can I split by 1 or more occurrences of a delimiter in Python?
  29. Python split string in moving window
  30. Splitting a math expression string into tokens in Python

Leave a Comment