How to reliably guess the encoding between MacRoman, CP1252, Latin1, UTF-8, and ASCII

by

First, the easy cases:

ASCII

If your data contains no bytes above 0x7F, then it’s ASCII. (Or a 7-bit ISO646 encoding, but those are very obsolete.)

UTF-8

If your data validates as UTF-8, then you can safely assume it is UTF-8. Due to UTF-8’s strict validation rules, false positives are extremely rare.

ISO-8859-1 vs. windows-1252

The only difference between these two encodings is that ISO-8859-1 has the C1 control characters where windows-1252 has the printable characters €‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ. I’ve seen plenty of files that use curly quotes or dashes, but none that use C1 control characters. So don’t even bother with them, or ISO-8859-1, just detect windows-1252 instead.

That now leaves you with only one question.

How do you distinguish MacRoman from cp1252?

This is a lot trickier.

Undefined characters

The bytes 0x81, 0x8D, 0x8F, 0x90, 0x9D are not used in windows-1252. If they occur, then assume the data is MacRoman.

Identical characters

The bytes 0xA2 (¢), 0xA3 (£), 0xA9 (©), 0xB1 (±), 0xB5 (µ) happen to be the same in both encodings. If these are the only non-ASCII bytes, then it doesn’t matter whether you choose MacRoman or cp1252.

Statistical approach

Count character (NOT byte!) frequencies in the data you know to be UTF-8. Determine the most frequent characters. Then use this data to determine whether the cp1252 or MacRoman characters are more common.

For example, in a search I just performed on 100 random English Wikipedia articles, the most common non-ASCII characters are ·•–é°®’èö—. Based on this fact,

  • The bytes 0x92, 0x95, 0x96, 0x97, 0xAE, 0xB0, 0xB7, 0xE8, 0xE9, or 0xF6 suggest windows-1252.
  • The bytes 0x8E, 0x8F, 0x9A, 0xA1, 0xA5, 0xA8, 0xD0, 0xD1, 0xD5, or 0xE1 suggest MacRoman.

Count up the cp1252-suggesting bytes and the MacRoman-suggesting bytes, and go with whichever is greatest.

More Related Contents:

  1. Rename Duplicate file name to another name in multiple folders [closed]
  2. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  3. Spark Error – Unsupported class file major version
  4. Java: How to detect (and change?) encoding of System.console?
  5. Run c/java/python with PHP? [duplicate]
  6. How to loop my game back to the start
  7. French and Chinese characters are not appearing correctly
  8. How to Find the Default Charset/Encoding in Java?
  9. How to set JAVA_HOME environment variable on Mac OS X 10.9?
  10. What is the default encoding of the JVM?
  11. Removing Java 8 JDK from Mac
  12. Class JavaLaunchHelper is implemented in both … libinstrument.dylib. One of the two will be used. Which one is undefined
  13. How to set request encoding in Tomcat?
  14. Printing my Mac’s serial number in java using Unix commands
  15. How to check if a String contains only ASCII?
  16. Cannot load R xlsx package on Mac OS 10.11
  17. JSON character encoding
  18. Guessing the encoding of text represented as byte[] in Java
  19. Java “Virtual Machine” vs. Python “Interpreter” parlance?
  20. JDBC character encoding
  21. How do I run Eclipse using Oracle’s new 1.7 JDK for the Mac?
  22. XML instance generation from XML schema (xsd) [closed]
  23. Add a dependency in Maven
  24. Choosing Java vs Python on Google App Engine
  25. Platform’s default charset on different platforms?
  26. Algorithm – How to delete duplicate elements in a list efficiently?
  27. Java PreparedStatement UTF-8 character problem
  28. JavaFX screencapture headless exception on OSX
  29. Convert .Jar to .Dmg
  30. Using NumPy and Cpython with Jython

Leave a Comment