How to convert a UTF-8 string into Unicode?

by

So the issue is that UTF-8 code unit values have been stored as a sequence of 16-bit code units in a C# string. You simply need to verify that each code unit is within the range of a byte, copy those values into bytes, and then convert the new UTF-8 byte sequence into UTF-16.

public static string DecodeFromUtf8(this string utf8String)
{
    // copy the string as UTF-8 bytes.
    byte[] utf8Bytes = new byte[utf8String.Length];
    for (int i=0;i<utf8String.Length;++i) {
        //Debug.Assert( 0 <= utf8String[i] && utf8String[i] <= 255, "the char must be in byte's range");
        utf8Bytes[i] = (byte)utf8String[i];
    }

    return Encoding.UTF8.GetString(utf8Bytes,0,utf8Bytes.Length);
}

DecodeFromUtf8("d\u00C3\u00A9j\u00C3\u00A0"); // déjà

This is easy, however it would be best to find the root cause; the location where someone is copying UTF-8 code units into 16 bit code units. The likely culprit is somebody converting bytes into a C# string using the wrong encoding. E.g. Encoding.Default.GetString(utf8Bytes, 0, utf8Bytes.Length).

Alternatively, if you’re sure you know the incorrect encoding which was used to produce the string, and that incorrect encoding transformation was lossless (usually the case if the incorrect encoding is a single byte encoding), then you can simply do the inverse encoding step to get the original UTF-8 data, and then you can do the correct conversion from UTF-8 bytes:

public static string UndoEncodingMistake(string mangledString, Encoding mistake, Encoding correction)
{
    // the inverse of `mistake.GetString(originalBytes);`
    byte[] originalBytes = mistake.GetBytes(mangledString);
    return correction.GetString(originalBytes);
}

UndoEncodingMistake("d\u00C3\u00A9j\u00C3\u00A0", Encoding(1252), Encoding.UTF8);

More Related Contents:

  1. How can I transform string to UTF-8 in C#?
  2. Characters in string changed after downloading HTML from the internet
  3. How to compare Unicode characters that “look alike”?
  4. MySQL C# Text Encoding Problems
  5. Why is the length of this string longer than the number of characters in it?
  6. Parsing through Arabic / RTL text from left to right
  7. Identify if a string is a number
  8. Easiest way to read from and write to files
  9. Converting string to title case
  10. Most efficient way to remove special characters from string
  11. Serialize an object to string
  12. What does $ mean before a string?
  13. Converting string to double in C#
  14. Find text in string with C#
  15. Is there a C# case insensitive equals operator?
  16. Is string in array?
  17. Remove text in-between delimiters in a string (using a regex?)
  18. Pass C# string to C++ and pass C++ result (string, char*.. whatever) to C#
  19. C programming: How to program for Unicode?
  20. How do I capitalize first letter of first name and last name in C#?
  21. Truncate string on whole words in .NET C#
  22. Display Float as String with at Least 1 Decimal Place
  23. Delete last char of string
  24. Why is \ character being doubled in my @ string?
  25. string.Join on a List or other type
  26. What does the @ prefix do on string literals in C#
  27. How to sort list of Ip Addresses using c#
  28. Regular expression to find and remove duplicate words
  29. Inheriting and overriding functions of a std::string?
  30. splitting a string based on multiple char delimiters

Leave a Comment