Handling special characters [closed]

by

The problem here is that you’ve stored a UTF-8 string to a different encoding in your database – probably the Windows-1252 code page (CP2152). As a result the UTF-8 character ’ represented by the byte sequence E2 80 99 is translated into the CP2152 single-byte characters ’. This was all explained to your previously in this answer, which also gives a solution to your current problem.

In order to get back to the original UTF-8 encoding you will need to take the string returned from your database and correct it with the following code:

public static string UTF8From1252(string source)
{
    // get original UTF-8 bytes from CP1252-encoded string
    byte[] bytes = System.Text.Encoding.GetEncoding("windows-1252").GetBytes(source);
    return System.Text.Encoding.UTF8.GetString(bytes);
}

This highlights the fact that it is vital to use the correct encoding at all times when using the GetBytes method.

It is important to note that the reverse of this transformation is not always possible, since there are gaps in the CP2152 code space – values that will be discarded or altered during conversion from byte values.

The hex values for these gaps are: 81 8D 8F 90 9D.

Unfortunately these values are present in various UTF-8 encodings, such as ” (E2 80 9D). If you have one of these values in your database then it will not load correctly. Depending on how you did the first stage conversion the third byte may be lost or corrupted in the database, in which case you cannot retrieve it.

More Related Contents:

  1. Encode string URL without using HttpServerUtility [duplicate]
  2. How can I detect the encoding/codepage of a text file
  3. Convert a Unicode string to an escaped ASCII string
  4. Converting string to byte array in C#
  5. Determine a string’s encoding in C#
  6. How to achieve Base64 URL safe encoding in C#?
  7. Effective way to find any file’s Encoding
  8. How can I convert a hex string to a byte array? [duplicate]
  9. C# Convert string from UTF-8 to ISO-8859-1 (Latin1) H
  10. How to detect the character encoding of a text file?
  11. How can I transform string to UTF-8 in C#?
  12. C#: Class for decoding Quoted-Printable encoding?
  13. Strip the byte order mark from string in C#
  14. Does C# have an equivalent to JavaScript’s encodeURIComponent()?
  15. How do I decode a base64 encoded string?
  16. How do you remove invalid hexadecimal characters from an XML-based data source prior to constructing an XmlReader or XPathDocument that uses the data?
  17. How to get ASCII value of string in C#
  18. Signing and verifying signatures with RSA C#
  19. C# Help reading foreign characters using StreamReader
  20. Is there a standard way to encode a .NET string into JavaScript string for use in MS Ajax?
  21. Encoding trouble with HttpWebResponse
  22. How do I write out a text file in C# with a code page other than UTF-8?
  23. Encoding.GetEncoding can’t work in UWP app
  24. What’s the difference between UTF8/UTF16 and Base64 in terms of encoding
  25. Encoding parameters for a URL
  26. How to put an encoding attribute to xml other that utf-16 with XmlWriter?
  27. ASCIIEncoding In Windows Phone 7
  28. Convert UTF-8 to base64 string
  29. System.Net.Mail and =?utf-8?B?XXXXX…. Headers
  30. Cyrillic encoding in C#

Leave a Comment