Here we run into the little used locale properties. If you output your string as a string (rather than raw data) you can get the locale to do the appropriate conversion auto-magically. N.B.This code does not take into account edianness of the wchar_t character. #include <locale> #include <fstream> #include <iostream> // See Below for the … Read more
Depends what you mean by ‘support’. You can certainly put non-UCS-2 characters in a JS string using surrogates, and browsers will display them if they can. But, each item in a JS string is a separate UTF-16 code unit. There is no language-level support for handling full characters: all the standard String members (length, split, … Read more
Wow. On the one hand I’m thrilled to know that university courses are teaching to the reality that character encodings are hard work, but actually knowing the UTF-8 encoding rules sounds like expecting a lot. (Will it help students pass the Turkey test?) The clearest description I’ve seen so far for the rules to encode … Read more
Simply use the “utf-8-sig” codec: fp = open(“file.txt”) s = fp.read() u = s.decode(“utf-8-sig”) That gives you a unicode string without the BOM. You can then use s = u.encode(“utf-8”) to get a normal UTF-8 encoded string back in s. If your files are big, then you should avoid reading them all into memory. The … Read more
I’ve been struggling with this problem for a while, and just discovered (for me) a perfect solution: $ git config –global diff.tool vimdiff # or merge.tool to get merging too! $ git difftool commit1 commit2 git difftool takes the same arguments as git diff would, but runs a diff program of your choice instead of … Read more
UTF-8 has an advantage in the case where ASCII characters represent the majority of characters in a block of text, because UTF-8 encodes these into 8 bits (like ASCII). It is also advantageous in that a UTF-8 file containing only ASCII characters has the same encoding as an ASCII file. UTF-16 is better where ASCII … Read more
Don’t worry about that. According to the same information source: this library component should be retired to Annex D, along side , until a suitable replacement is standardized. So, you can still use it until a new standardized, more-secure version is done.
Java uses UTF-16 for the internal text representation The representation for String and StringBuilder etc in Java is UTF-16 https://docs.oracle.com/javase/8/docs/technotes/guides/intl/overview.html How is text represented in the Java platform? The Java programming language is based on the Unicode character set, and several libraries implement the Unicode standard. The primitive data type char in the Java programming … Read more
Why do we need Unicode? In the (not too) early days, all that existed was ASCII. This was okay, as all that would ever be needed were a few control characters, punctuation, numbers and letters like the ones in this sentence. Unfortunately, today’s strange world of global intercommunication and social media was not foreseen, and … Read more
The term “surrogate pair” refers to a means of encoding Unicode characters with high code-points in the UTF-16 encoding scheme. In the Unicode character encoding, characters are mapped to values between 0x0 and 0x10FFFF. Internally, Java uses the UTF-16 encoding scheme to store strings of Unicode text. In UTF-16, 16-bit (two-byte) code units are used. … Read more