Javascript unicode string, chinese character but no punctuation

by

You can see the relevant blocks at http://www.unicode.org/reports/tr38/#BlockListing or http://www.unicode.org/charts/ .

If you are excluding compatibility characters (ones which should no longer be used), as well as strokes, radicals, and Enclosed CJK Letters and Months, the following ought to cover it (I’ve added the individual JavaScript equivalent expressions afterward):

  • CJK Unified Ideographs (4E00-9FCC) [\u4E00-\u9FCC]
  • CJK Unified Ideographs Extension A (3400-4DB5) [\u3400-\u4DB5]
  • CJK Unified Ideographs Extension B (20000-2A6D6) [\ud840-\ud868][\udc00-\udfff]|\ud869[\udc00-\uded6]
  • CJK Unified Ideographs Extension C (2A700-2B734) \ud869[\udf00-\udfff]|[\ud86a-\ud86c][\udc00-\udfff]|\ud86d[\udc00-\udf34]
  • CJK Unified Ideographs Extension D (2B840-2B81D) \ud86d[\udf40-\udfff]|\ud86e[\udc00-\udc1d]
  • 12 characters within the CJK Compatibility Ideographs (F900-FA6D/FA70-FAD9) but which are actually CJK unified ideographs [\uFA0E\uFA0F\uFA11\uFA13\uFA14\uFA1F\uFA21\uFA23\uFA24\uFA27-\uFA29]

…so, a regex to grab the Chinese characters would be:

/[\u4E00-\u9FCC\u3400-\u4DB5\uFA0E\uFA0F\uFA11\uFA13\uFA14\uFA1F\uFA21\uFA23\uFA24\uFA27-\uFA29]|[\ud840-\ud868][\udc00-\udfff]|\ud869[\udc00-\uded6\udf00-\udfff]|[\ud86a-\ud86c][\udc00-\udfff]|\ud86d[\udc00-\udf34\udf40-\udfff]|\ud86e[\udc00-\udc1d]/

Due in fact to the many CJK (Chinese-Japanese-Korean) characters, Unicode was expanded to handle more characters beyond the “Basic Multilingual Plane” (called “astral” characters), and since the CJK Unified Ideographs extensions B-D are examples of such astral characters, those extensions have ranges that are more complicated because they have to be encoded using surrogate pairs in UTF-16 systems like JavaScript. A surrogate pair consists of a high surrogate and a low surrogate, neither of which is valid by itself but when joined together form an actual single character despite their string length being 2).

While it would probably be easier for replacement purposes to express this as the non-Chinese characters (to replace them with the empty string), I provided the expression for the Chinese characters instead so that it would be easier to track in case you needed to add or remove from the blocks.

Update September 2017

As of ES6, one may express the regular expressions without resorting to surrogates by using the “u” flag along with the code point inside of the new escape sequence with brackets, e.g., /^[\u{20000}-\u{2A6D6}]*$/u for “CJK Unified Ideographs Extension B”.

Note that Unicode too has progressed to include “CJK Unified Ideographs Extension E” ([\u{2B820}-\u{2CEAF}]) and “CJK Unified Ideographs Extension F” ([\u{2CEB0}-\u{2EBEF}]).

For ES2018, it appears that Unicode property escapes will be able to simplify things even further. Per http://2ality.com/2017/07/regexp-unicode-property-escapes.html , it looks like will be able to do:

/^(\p{Block=CJK Unified Ideographs}|\p{Block=CJK Unified Ideographs Extension A}|\p{Block=CJK Unified Ideographs Extension B}|\p{Block=CJK Unified Ideographs Extension C}|\p{Block=CJK Unified Ideographs Extension D}|\p{Block=CJK Unified Ideographs Extension E}|\p{Block=CJK Unified Ideographs Extension F}|[\uFA0E\uFA0F\uFA11\uFA13\uFA14\uFA1F\uFA21\uFA23\uFA24\uFA27-\uFA29])+$/u

And as the shorter aliases from http://unicode.org/Public/UNIDATA/PropertyAliases.txt and http://unicode.org/Public/UNIDATA/PropertyValueAliases.txt can also be used for these blocks, you could shorten this to the following (and changing underscores to spaces or casing apparently too if desired):
/^(\p{Blk=CJK}|\p{Blk=CJK_Ext_A}|\p{Blk=CJK_Ext_B}|\p{Blk=CJK_Ext_C}|\p{Blk=CJK_Ext_D}|\p{Blk=CJK_Ext_E}|\p{Blk=CJK_Ext_F}|[\uFA0E\uFA0F\uFA11\uFA13\uFA14\uFA1F\uFA21\uFA23\uFA24\uFA27-\uFA29])+$/u

And if we wanted to improve readability, we could document the falsely labeled compatibility characters using named capture groups (see http://2ality.com/2017/05/regexp-named-capture-groups.html ):

/^(\p{Blk=CJK}|\p{Blk=CJK_Ext_A}|\p{Blk=CJK_Ext_B}|\p{Blk=CJK_Ext_C}|\p{Blk=CJK_Ext_D}|\p{Blk=CJK_Ext_E}|\p{Blk=CJK_Ext_F}|(?<CJKFalseCompatibilityUnifieds>[\uFA0E\uFA0F\uFA11\uFA13\uFA14\uFA1F\uFA21\uFA23\uFA24\uFA27-\uFA29]))+$/u

And as it looks per http://unicode.org/reports/tr44/#Unified_Ideograph like the “Unified_Ideograph” property (alias “UIdeo”) covers all of our unified ideographs and excluding symbols/punctuation and compatibility characters, if you don’t need to pick and choose out of the above, the following may be all you need:

/^\p{Unified_Ideograph=yes}*$/u

or in shorthand:

/^\p{UIdeo=y}*$/u

More Related Contents:

  1. How can I use Unicode-aware regular expressions in JavaScript?
  2. How to count string occurrence in string?
  3. Regular Expression to get a string between parentheses in Javascript
  4. Concrete JavaScript regular expression for accented characters (diacritics)
  5. Javascript RegExp + Word boundaries + unicode characters
  6. Check for special characters in string
  7. Get Substring between two characters using javascript
  8. How to find indices of all occurrences of one string in another in JavaScript?
  9. Use dynamic (variable) string as regex pattern in JavaScript
  10. Regular expression which matches a pattern, or is an empty string
  11. utf-8 word boundary regex in javascript
  12. Javascript – return string between square brackets
  13. javascript regular expression to check for IP addresses
  14. Passing regex modifier options to RegExp object
  15. JavaScript regular expression for Unicode emoji
  16. How to extract base URL from a string in JavaScript?
  17. Matching accented characters with Javascript regexes
  18. Wrap tags around http text
  19. match() returns array with two matches when I expect one match
  20. Split string on the first white space occurrence
  21. Split a string based on multiple delimiters
  22. Javascript string comparison fails when comparing unicode characters
  23. Insert space before capital letters
  24. Why can’t I use accented characters next to a word boundary?
  25. Use RegExp to match a parenthetical number then increment it
  26. JavaScript: indexOf vs. Match when Searching Strings?
  27. Removing backslashes from strings in javascript
  28. javascript and string manipulation w/ utf-16 surrogate pairs
  29. Why this regex is not working for german words?
  30. Find and replace nth occurrence of [bracketed] expression in string

Leave a Comment