What is the most accurate encoding detector? [closed]

by

I’ve checked juniversalchardet and ICU4J on some CSV files, and the results are inconsistent:
juniversalchardet had better results:

  • UTF-8: Both detected.
  • Windows-1255: juniversalchardet detected when it had enough hebrew letters, ICU4J still thought it was ISO-8859-1. With even more hebrew letters, ICU4J detected it as ISO-8859-8 which is the other hebrew encoding(and so the text was OK).
  • SHIFT_JIS(Japanese): juniversalchardet detected and ICU4J thought it was ISO-8859-2.
  • ISO-8859-1: detected by ICU4J, not supported by juniversalchardet.

So one should consider which encodings he will most likely have to deal with.
In the end I chose ICU4J.

Notice that ICU4J is still maintained.

Also notice that you may want to use ICU4J, and in case that it returns null because it didn’t succeed, try to use juniversalchardet. Or the opposite.

AutoDetectReader of Apache Tika does exactly this – first tries to use HtmlEncodingDetector, then UniversalEncodingDetector(which is based on juniversalchardet), and then tries Icu4jEncodingDetector(based on ICU4J).

More Related Contents:

  1. Setting the default Java character encoding
  2. Java : How to determine the correct charset encoding of a stream
  3. How to convert Strings to and from UTF8 byte arrays in Java
  4. Spring MVC UTF-8 Encoding
  5. Who sets response content-type in Spring MVC (@ResponseBody)
  6. Encoding conversion in java
  7. How do I convert between ISO-8859-1 and UTF-8 in Java?
  8. How to add a UTF-8 BOM in Java?
  9. HttpServletRequest – setCharacterEncoding seems to do nothing
  10. JSON character encoding
  11. UTF-8 charset doesn’t work with javax.mail
  12. Java Unicode encoding
  13. Converting byte array to String (Java)
  14. Why does Java’s String.getBytes() uses “ISO-8859-1”
  15. Display special characters using System.out.println
  16. ISO-8859-1 encoding and binary data preservation
  17. Default character encoding for java console output
  18. What is the character encoding of String in Java?
  19. UTF-8 encoding problem in Spring MVC
  20. In Java, how to find if first character in a string is upper case without regex
  21. Unable to change charset from ISO-8859-1 to UTF-8 in glassfish 3.1
  22. How do I convert a byte array with null terminating character to a String in Java?
  23. All inclusive Charset to avoid “java.nio.charset.MalformedInputException: Input length = 1”?
  24. UTF-8 text (Hindi) not getting displayed on Browser window or Eclipse console
  25. JsonParseException : Illegal unquoted character ((CTRL-CHAR, code 10)
  26. Java: Converting String to and from ByteBuffer and associated problems
  27. First character of the reading from the text file :  [duplicate]
  28. Java compiler platform file encoding problem
  29. Detect the URI encoding automatically in Tomcat
  30. How to correctly compute the length of a String in Java?

Leave a Comment