What’s the correct regex range for javascript’s regexes to match all the non word characters in any script?

by

Generic solution

Mathias Bynens suggests to follow the UTS18 recommendation and thus a Unicode-aware \W will look like:

[^\p{Alphabetic}\p{Mark}\p{Decimal_Number}\p{Connector_Punctuation}\p{Join_Control}]

Please note the comment for the suggested Unicode property class combination:

This is only an approximation to Word Boundaries (see b below). The
Connector Punctuation is added in for programming language
identifiers, thus adding “_” and similar characters.

More considerations

The \w construct (and thus its \W counterpart), when matching in a Unicode-aware context, matches similar, but somewhat different set of characters across regex engines.

For example, here is Non-word character: \W .NET definition: [^\p{Ll}\p{Lu}\p{Lt}\p{Lo}\p{Nd}\p{Mn}\p{Pc}\p{Lm}], where \p{Ll}\p{Lu}\p{Lt}\p{Lo} can be contracted to a sheer \p{L} and the pattern is thus equal to [^\p{L}\p{Nd}\p{Mn}\p{Pc}].

In Android (see documentation), [^\p{Alpha}\p{gc=Mn}\p{gc=Me}\p{gc=Mc}\p{Digit}\p{gc=Pc}\p{IsJoin_Control}], where \p{gc=Mn}\p{gc=Me}\p{gc=Mc} can be just written as \p{M}.

In PHP PCRE, \W matches [^\p{L}\p{N}_].

Rexegg cheat sheet defines Python 3 \w as “Unicode letter, ideogram, digit, or underscore“, i.e. [\p{L}\p{Mn}\p{Nd}_].

You may roughly decompose \W as [^\p{L}\p{N}\p{M}\p{Pc}]:

/[^\p{L}\p{N}\p{M}\p{Pc}]/gu

where

  • [^ – is the start of the negated character class that matches a single char other than:
    • \p{L} – any Unicode letter
    • \p{N} – any Unicode digit
    • \p{M} – a diacritic mark
    • \p{Pc} – a connector punctuation symbol
  • ] – end of the character class.

Note it is \p{Pc} class that matches an underscore.

NOTE that \p{Alphabetic} (\p{Alpha}) includes all letters matched by \p{L}, plus letter numbers matched by \p{Nl} (e.g. Ⅻ – a character for the roman number 12), plus some other symbols matched with \p{Other_Alphabetic} (\p{OAlpha}).

Other variations:

  • /[^\p{L}0-9_]/gu – to just use \W that is aware of Unicode letters only
  • /[^\p{L}\p{N}_]/gu – (PCRE \W style) to just use \W that is aware of Unicode letters and digits only.

Note that Java’s (?U)\W will match a mix of what \W matches in PCRE, Python and .NET.

More Related Contents:

  1. Find list of strings that matches a pattern that start with a special character « and end with a special character »
  2. Need Pattern for password field?
  3. Javascript Regex for decimal [duplicate]
  4. Regex to get string between curly braces
  5. Detect URLs in text with JavaScript
  6. Regular Expression to get a string between parentheses in Javascript
  7. What is the meaning of the ‘g’ flag in regular expressions?
  8. What is the difference between RegExp’s exec() function and String’s match() function?
  9. Is there a version of JavaScript’s String.indexOf() that allows for regular expressions?
  10. split string only on first instance of specified character
  11. Remove HTML Tags in Javascript with Regex
  12. javascript regex split produces too many items
  13. Remove HTML comments with Regex, in Javascript
  14. Create RegExps on the fly using string variables
  15. Regular expression for URL validation (in JavaScript)
  16. How to replace last occurrence of characters in a string using javascript
  17. Validating user’s UTF-8 name in Javascript
  18. Javascript fuzzy search that makes sense
  19. Regex for number with decimals and thousand separator
  20. Javascript regular expression to validate URL
  21. Regex to extract substring, returning 2 results for some reason
  22. Split a string by whitespace, keeping quoted segments, allowing escaped quotes
  23. Extract all email addresses from bulk text using jquery
  24. Split string on the first white space occurrence
  25. Regular expression to remove a file’s extension
  26. Regex to split camel case
  27. JavaScript remove ZERO WIDTH SPACE (unicode 8203) from string
  28. What’s different between Python and Javascript regular expressions?
  29. Extract keyword from Google search in Javascript
  30. Regex matching emoticons

Leave a Comment