“Fix” String encoding in Java

by

As there seems to be some confusion on whether this is possible or not I think I’ll need to provide an extensive example.

The question claims that the (initial) input is a byte[] that contains Windows-1252 encoded data. I’ll call that byte[] ib (for “initial bytes”).

For this example I’ll choose the German word “Bär” (meaning bear) as the input:

byte[] ib = new byte[] { (byte) 0x42, (byte) 0xE4, (byte) 0x72 };
String correctString = new String(ib, "Windows-1252");
assert correctString.charAt(1) == '\u00E4'; //verify that the character was correctly decoded.

(If your JVM doesn’t support that encoding, then you can use ISO-8859-1 instead, because those three letters (and most others) are at the same position in those two encodings).

The question goes on to state that some other code (that is outside of our influence) already converted that byte[] to a String using the UTF-8 encoding (I’ll call that String is for “input String”). That String is the only input that is available to achieve our goal (if ib were available, it would be trivial):

String is = new String(ib, "UTF-8");
System.out.println(is);

This obviously produces the incorrect output “B�”.

The goal would be to produce ib (or the correct decoding of that byte[]) with only is available.

Now some people claim that getting the UTF-8 encoded bytes from that is will return an array with the same values as the initial array:

byte[] utf8Again = is.getBytes("UTF-8");

But that returns the UTF-8 encoding of the two characters B and � and definitely returns the wrong result when re-interpreted as Windows-1252:

System.out.println(new String(utf8Again, "Windows-1252");

This line produces the output “B�”, which is totally wrong (it is also the same output that would be the result if the initial array contained the non-word “Bür” instead).

So in this case you can’t undo the operation, because some information was lost.

There are in fact cases where such mis-encodings can be undone. It’s more likely to work, when all possible (or at least occuring) byte sequences are valid in that encoding. Since UTF-8 has several byte sequences that are simply not valid values, you will have problems.

More Related Contents:

  1. Java URL encoding of query string parameters
  2. How to convert a string with Unicode encoding to a string of letters
  3. How to Find the Default Charset/Encoding in Java?
  4. What is the Java’s internal represention for String? Modified UTF-8? UTF-16?
  5. Reading UTF-8 – BOM marker
  6. What is the default encoding of the JVM?
  7. Best way to encode text data for XML in Java?
  8. How to encode a WAV to a mp3 on a Android device
  9. How to configure encoding in Maven?
  10. Java properties UTF-8 encoding in Eclipse
  11. java.sql.SQLException: Incorrect string value: ‘\xF0\x9F\x91\xBD\xF0\x9F…’
  12. “unmappable character for encoding” warning in Java
  13. “Unmappable character for encoding UTF-8” error
  14. Guessing the encoding of text represented as byte[] in Java
  15. IntelliJ IDEA incorrect encoding in console output
  16. File was loaded in the wrong encoding:’UTF-8′ in IntelliJ IDEA
  17. Java 8 UTF-8 encoding issue (java bug?)
  18. Java: How to detect (and change?) encoding of System.console?
  19. How can I safely encode a string in Java to use as a filename?
  20. Check if a String is valid UTF-8 encoded in Java
  21. cyrillic in windows Console(java) System.out.println();
  22. Byte and char conversion in Java
  23. HttpServletRequest UTF-8 Encoding [duplicate]
  24. How to use Special Chars in Java/Eclipse
  25. Why is Java BufferedReader() not reading Arabic and Chinese characters correctly?
  26. Unable to print russian characters
  27. How to determine if a String contains invalid encoded characters
  28. How to force a jar to use(or the jvm that jar runs in) utf-8 instead of the system’s default encoding
  29. Producing valid XML with Java and UTF-8 encoding
  30. Java Encode file to Base64 string To match with other encoded string

Leave a Comment