You don’t get the character in the editor, because you can’t find it in text editors. #FEFF or #FFFE are so-called byte-order marks. They are a Microsoft invention to tell in a Unicode file, in which order multi-byte characters are stored. To get rid of it, tell your editor to save the file either as … Read more
There’s no ambiguity here: RFC3986 says no, that is, URIs cannot contain unicode characters, only ASCII. An entirely different matter is how browsers represent encoded characters when displaying a URI, for example some browsers will display a space in a URL instead of ‘%20’. This is how IDN works too: punycoded strings are encoded and … Read more
Is the new standard more open w.r.t to Unicode? With respect to allowing universal character names in identifiers the answer is no; UCNs were allowed in identifiers back in C99 and C++98. However compilers did not implement that particular requirement until recently. Clang 3.3 I think introduces support for this and GCC has had an … Read more
You could change the $OutputEncoding variable before writing to the file. The other option is not to use the > operator, but instead pipe directly to Out-File and use the -Encoding parameter.
As zawhtut mentioned, this page has a reference for several unicode ranges. To summarize the ranges: Japanese-style punctuation ( 3000 – 303f) Hiragana ( 3040 – 309f) Katakana ( 30a0 – 30ff) Full-width roman characters and half-width katakana ( ff00 – ffef) CJK unifed ideographs – Common and uncommon kanji ( 4e00 – 9faf)
Emoji are now the most common non-BMP characters by far. 😂, otherwise known as U+1F602 FACE WITH TEARS OF JOY, is the most common one on Twitter’s public stream. It occurs more frequently than the tilde!
I found this tutorial very useful. Yes it is partially an ad for another Unicode Control Suite, but it has a lot of information about how to do it yourself and what issues are involved. EDIT I knew I had way more on this stored in my bookmarks. First of all there is an article … Read more
I am asking for the count of all the possible valid combinations in Unicode with explanation. 1,111,998: 17 planes × 65,536 characters per plane – 2048 surrogates – 66 noncharacters Note that UTF-8 and UTF-32 could theoretically encode much more than 17 planes, but the range is restricted based on the limitations of the UTF-16 … Read more
Each byte starts with a few bits that tell you whether it’s a single byte code-point, a multi-byte code point, or a continuation of a multi-byte code point. Like this: 0xxx xxxx A single-byte US-ASCII code (from the first 127 characters) The multi-byte code-points each start with a few bits that essentially say “hey, you … Read more
Check out Markus Kuhn’s UTF-8 decoder stress test