Big-Endian (BE) / Little-Endian (LE) are two ways to organize multi-byte words. For example, when using two bytes to represent a character in UTF-16, there are two ways to represent the character 0x1234 as a string of bytes (0x00-0xFF): Byte Index: 0 1 ——————— Big-Endian: 12 34 Little-Endian: 34 12 In order to decide if … Read more
This is not a PowerShell deficiency. It is a deficiency with the Windows console subsystem which PowerShell.exe uses. The console subsystem does not support Unicode but code pages instead which dates back to the DOS days. The PowerShell V2 fix is provided via the PowerShell Integrated Scripting Environment or PowerShell_ISE.exe. This is a graphical app … Read more
From what I can gather in the documentation, they want you to get Character values from a String because it gives context. Is this Character encoded with UTF8, UTF16, or 21-bit code points (scalars)? If you look at how a Character is defined in the Swift framework, it is actually an enum value. This is … Read more
The UTF-8 BOM is a sequence of bytes at the start of a text stream (0xEF, 0xBB, 0xBF) that allows the reader to more reliably guess a file as being encoded in UTF-8. Normally, the BOM is used to signal the endianness of an encoding, but since endianness is irrelevant to UTF-8, the BOM is … Read more
The easiest way is to just convert the text file to utf-8 and pipe that to grep: iconv -f utf-16 -t utf-8 file.txt | grep query I tried to do the opposite (convert my query to utf-16) but it seems as though grep doesn’t like that. I think it might have to do with endianness, … Read more
There is a plain-text database of information about every Unicode character available from the Unicode Consortium; the format is described in Unicode Annex #44. The primary information is contained in UnicodeData.txt. Open and close punctuation characters are denoted with Ps (punctuation start) and Pe (punctuation end) in the General_Category field (the third field, delimited by … Read more
Yes, the zero byte in UTF8 is code point 0, NUL. There is no other Unicode code point that will be encoded in UTF8 with a zero byte anywhere within it. The possible code points and their UTF8 encoding are: Range Encoding Binary value —————– ——– ————————– U+000000-U+00007f 0xxxxxxx 0xxxxxxx U+000080-U+0007ff 110yyyxx 00000yyy xxxxxxxx 10xxxxxx … Read more
According to the code charts, U+0660 .. U+0669 are ARABIC-INDIC DIGIT values 0 through 9, while U+06F0 .. U+06F9 are EXTENDED ARABIC-INDIC DIGIT values 0 through 9. In the Unicode 3.0 book (5.2 is the current version, but these things don’t change much once set), the U+066n series of glyphs are marked ‘Arabic-Indic digits’ and … Read more
You don’t get the character in the editor, because you can’t find it in text editors. #FEFF or #FFFE are so-called byte-order marks. They are a Microsoft invention to tell in a Unicode file, in which order multi-byte characters are stored. To get rid of it, tell your editor to save the file either as … Read more
There’s no ambiguity here: RFC3986 says no, that is, URIs cannot contain unicode characters, only ASCII. An entirely different matter is how browsers represent encoded characters when displaying a URI, for example some browsers will display a space in a URL instead of ‘%20’. This is how IDN works too: punycoded strings are encoded and … Read more