I can’t say I can rely on mb_detect_encoding(). I had some freaky false positives a while back. The most universal way I found to work well in every case was: if (preg_match(‘!!u’, $string)) { // This is UTF-8 } else { // Definitely not UTF-8 }
In Python 2, Unicode strings may contain both unicode and bytes: No, they may not. They contain Unicode characters. Within the original string, \xd0 is not a byte that’s part of a UTF-8 encoding. It is the Unicode character with code point 208. u’\xd0′ == u’\u00d0′. It just happens that the repr for Unicode strings … Read more
git is dumping out raw bytes. In this case, it doesn’t care what your file’s encoding is. The highlighted <F6> you’re seeing is coming from less, which is presumably configured as your PAGER. Try setting: LESSCHARSET=UTF-8
Your “bad” output is UTF-8 displayed as CP1252. On Windows, many editors assume the default ANSI encoding (CP1252 on US Windows) instead of UTF-8 if there is no byte order mark (BOM) character at the start of the file. While a BOM is meaningless to the UTF-8 encoding, its UTF-8-encoded presence serves as a signature … Read more
There is a GNU tool recode which you can also use on Windows. E.g. recode utf16..utf8 text.txt
Here’s an implementation based on the UTF-8 specification: public class Utf8LenCounter { public static int length(CharSequence sequence) { int count = 0; for (int i = 0, len = sequence.length(); i < len; i++) { char ch = sequence.charAt(i); if (ch <= 0x7F) { count++; } else if (ch <= 0x7FF) { count += 2; … Read more
php does try to spew an error, but only if you turn display_errors off. This is odd because the display_errors setting is only meant to control whether or not errors are printed to standard output, not whether or not an error is triggered. I want to emphasize that when you have display_errors on, even though … Read more
preg_replace(‘/^[\pZ\pC]+|[\pZ\pC]+$/u’,”,$str);
Have you tried adding SET NAMES ‘utf8′; to your sql dump? The thing with utf8 or encodings in general is that in order to be successfull, you have to make sure that: the file is encoded utf8 without signature the default encoding of the mysql server is set to utf8 the connection is utf8 (that’s … Read more
If you need MySQL to support 4-byte UTF-8 characters (which is normally considered part of UTF-8), you need to use the character set utf8mb4, not utf8. utf8mb4 was first supported in MySQL 5.5.3.