Guessing the encoding of text represented as byte[] in Java

by

The following method solves the problem using juniversalchardet, which is a Java port of Mozilla’s encoding detection library.

public static String guessEncoding(byte[] bytes) {
    String DEFAULT_ENCODING = "UTF-8";
    org.mozilla.universalchardet.UniversalDetector detector =
        new org.mozilla.universalchardet.UniversalDetector(null);
    detector.handleData(bytes, 0, bytes.length);
    detector.dataEnd();
    String encoding = detector.getDetectedCharset();
    detector.reset();
    if (encoding == null) {
        encoding = DEFAULT_ENCODING;
    }
    return encoding;
}

The code above has been tested and works as intented. Simply add juniversalchardet-1.0.3.jar to the classpath.

I’ve tested both juniversalchardet and jchardet. My general impression is that juniversalchardet provides the better detection accuracy and the nicer API of the two libraries.

More Related Contents:

  1. IntelliJ IDEA incorrect encoding in console output
  2. French and Chinese characters are not appearing correctly
  3. How to get UTF-8 working in Java webapps?
  4. Java : How to determine the correct charset encoding of a stream
  5. How to convert Strings to and from UTF8 byte arrays in Java
  6. Spring MVC UTF-8 Encoding
  7. How to Find the Default Charset/Encoding in Java?
  8. What is the default encoding of the JVM?
  9. How do I convert between ISO-8859-1 and UTF-8 in Java?
  10. Binary to text in Java
  11. Java properties UTF-8 encoding in Eclipse
  12. java.sql.SQLException: Incorrect string value: ‘\xF0\x9F\x91\xBD\xF0\x9F…’
  13. “unmappable character for encoding” warning in Java
  14. Java – Convert String to valid URI object
  15. “Unmappable character for encoding UTF-8” error
  16. UTF-8 charset doesn’t work with javax.mail
  17. File was loaded in the wrong encoding:’UTF-8′ in IntelliJ IDEA
  18. Java BufferedWriter object with utf-8
  19. Java: How to detect (and change?) encoding of System.console?
  20. Default character encoding for java console output
  21. UTF-8 encoding problem in Spring MVC
  22. Check if a String is valid UTF-8 encoded in Java
  23. Unable to change charset from ISO-8859-1 to UTF-8 in glassfish 3.1
  24. HttpServletRequest UTF-8 Encoding [duplicate]
  25. Should source code be saved in UTF-8 format
  26. Why is Java BufferedReader() not reading Arabic and Chinese characters correctly?
  27. Unable to print russian characters
  28. Java PreparedStatement UTF-8 character problem
  29. Producing valid XML with Java and UTF-8 encoding
  30. Java Encode file to Base64 string To match with other encoded string

Leave a Comment