R/regex with stringi/ICU: why is a ‘+’ considered a non-[:punct:] character?

by

POSIX character classes need to be wrapped inside of a character class, the correct form would be [[:punct:]]. Do not confuse the POSIX term “character class” with what is normally called a regex character class.

This POSIX named class in the ASCII range matches all non-controls, non-alphanumeric, non-space characters. 

ascii <- rawToChar(as.raw(0:127), multiple=T)
paste(ascii[grepl('[[:punct:]]', ascii)], collapse="")
# [1] "!\"#$%&'()*+,-./:;<=>?@[\\]^_`{|}~"

Although if a locale is in effect, it could alter the behavior of [[:punct:]] …

R Documentation ?regex states the following: Certain named classes of characters are predefined. Their interpretation depends on the locale (see locales); the interpretation is that of the POSIX locale.

The Open Group LC_TYPE definition for punct says:

Define characters to be classified as punctuation characters.

In the POSIX locale, neither the <space> nor any characters in classes alpha, digit, or cntrl shall be included.

In a locale definition file, no character specified for the keywords upper, lower, alpha, digit, cntrl, xdigit, or as the <space> shall be specified.

However, the stringi package seems to depend on ICU and locale is a fundamental concept in ICU.

Using the stringi package, I recommend using the Unicode Properties \p{P} and \p{S}.

  • \p{P} matches any kind of punctuation character. That is, it is missing nine of the characters that the POSIX class punct includes. This is because Unicode splits what POSIX considers to be punctuation into two categories, Punctuation and Symbols. This is where \p{S} comes into place …

    stri_replace_all_regex(string1, '[\\p{P}\\p{S}]', ' ')
# [1] "this is a test"            "this  is also a test"     
# [3] "this is the final  test"   "this is the final   test "

  • Or fallback to gsub from base R which handles this very well.

    gsub('[[:punct:]]', ' ', string1)
# [1] "this is a test"            "this  is also a test"     
# [3] "this is the final  test"   "this is the final   test "

More Related Contents:

  1. Overlapping matches in R
  2. Replace single backslash in R
  3. Remove all special characters from a string in R?
  4. Create new column in dataframe based on partial string matching other column
  5. Select rows from data.frame ending with a specific character string in R
  6. R split on delimiter (split) keep the delimiter (split)
  7. Find the location of a character in string
  8. Is there an R function to escape a string for regex characters
  9. Remove square brackets from a string vector
  10. Remove all characters before a period in a string
  11. How to measure similarity between strings?
  12. Extracting a string between other two strings in R
  13. How to remove all whitespace from a string?
  14. Extract info inside all parenthesis in R
  15. Regex group capture in R with multiple capture-groups
  16. Regular expression to match last number in a string
  17. Remove all text between two brackets
  18. Regular expression – starting and ending with a character string
  19. R dplyr: rename variables using string functions
  20. Regular Expression Match to test for a valid year
  21. Find numbers after specific text in a string with RegEx
  22. What is a regex to match ONLY an empty string?
  23. Splitting a string on the first space
  24. Regex empty string or email
  25. Test for numeric elements in a character string
  26. extract number after specific string
  27. Replace string only once with php preg_replace
  28. Extract last 4-digit number from a series in R using stringr
  29. Extracting text after last period in string [duplicate]
  30. How to prevent regmatches drop non matches?

Leave a Comment