javascript and string manipulation w/ utf-16 surrogate pairs

by

Javascript uses UCS-2 internally, which is not UTF-16. It is very difficult to handle Unicode in Javascript because of this, and I do not suggest attempting to do so.

As for what Twitter does, you seem to be saying that it is sanely counting by code point not insanely by code unit.

Unless you have no choice, you should use a programming language that actually supports Unicode, and which has a code-point interface, not a code-unit interface. Javascript isn’t good enough for that as you have discovered.

It has The UCS-2 Curse, which is even worse than The UTF-16 Curse, which is already bad enough. I talk about all this in OSCON talk, ๐Ÿ”ซ Unicode Support Shootout: ๐Ÿ‘ The Good, the Bad, & the (mostly) Ugly ๐Ÿ‘Ž.

Due to its horrible Curse, you have to hand-simulate UTF-16 with UCS-2 in Javascript, which is simply nuts.

Javascript suffers from all kinds of other terrible Unicode troubles, too. It has no support for graphemes or normalization or collation, all of which you really need. And its regexes are broken, sometimes due to the Curse, sometimes just because people got it wrong. For example, Javascript is incapable of expressing regexes like [๐’œ-๐’ต]. Javascript doesnโ€™t even support casefolding, so you canโ€™t write a pattern like /ฮฃฮคฮ™ฮ“ฮœฮ‘ฮฃ/i and have it correctly match ฯƒฯ„ฮนฮณฮผฮฑฯ‚.

You can try to use the XRegEXp plugin, but you wonโ€™t banish the Curse that way. Only changing to a language with Unicode support will do that, and ๐’ฅ๐’ถ๐“‹๐’ถ๐“ˆ๐’ธ๐“‡๐’พ๐“…๐“‰ just isnโ€™t one of those.

More Related Contents:

  1. JavaScript strings outside of the BMP
  2. Javascript unicode string, chinese character but no punctuation
  3. Javascript string comparison fails when comparing unicode characters
  4. Unicode string with diacritics split by chars
  5. Count the recurrence of a word
  6. How to get character array from a string?
  7. Check if a JavaScript string is a URL
  8. How to check if a string contains text from an array of substrings in JavaScript?
  9. Javascript RegExp + Word boundaries + unicode characters
  10. How to convert a string to an integer in JavaScript?
  11. How can I convert a comma-separated string to an array?
  12. How to set object property (of object property of..) given its string name in JavaScript?
  13. How can I insert a character after every n characters in javascript?
  14. How can I remove a character from a string using JavaScript?
  15. utf-8 word boundary regex in javascript
  16. Javascript – return string between square brackets
  17. Case insensitive string replacement in JavaScript?
  18. JavaScript string with new line – but not using \n
  19. Remove characters from a string [duplicate]
  20. How can I split a string containing emoji into an array?
  21. is the + operator less performant than StringBuffer.append()
  22. How to extract base URL from a string in JavaScript?
  23. Get the string representation of a DOM node
  24. Need a basename function in Javascript
  25. Split a string based on multiple delimiters
  26. Get object property name as a string
  27. Printing emojis with JavaScript and HTML
  28. Compare Strings Javascript Return %of Likely
  29. JavaScript: indexOf vs. Match when Searching Strings?
  30. Anagrams finder in javascript

Leave a Comment