Encoding trouble with HttpWebResponse

by

CharacterSet is “ISO-8859-1” by default, if it is not specified in server’s content type header (different from “charset” meta tag in HTML).
I compare HttpWebResponse.CharacterSet with charset attribute of HTML. If they are different – I use the charset as specified in HTML to re-read the page again, but with correct encoding this time.

See the code: 

    string strWebPage = "";
    // create request
    System.Net.WebRequest objRequest = System.Net.HttpWebRequest.Create(sURL);
    // get response
    System.Net.HttpWebResponse objResponse;
    objResponse = (System.Net.HttpWebResponse)objRequest.GetResponse();
    // get correct charset and encoding from the server's header
    string Charset = objResponse.CharacterSet;
    Encoding encoding = Encoding.GetEncoding(Charset);
    // read response
    using (StreamReader sr = 
           new StreamReader(objResponse.GetResponseStream(), encoding))
    {
        strWebPage = sr.ReadToEnd();
        // Close and clean up the StreamReader
        sr.Close();
    }

    // Check real charset meta-tag in HTML
    int CharsetStart = strWebPage.IndexOf("charset=");
    if (CharsetStart > 0)
    {
        CharsetStart += 8;
        int CharsetEnd = strWebPage.IndexOfAny(new[] { ' ', '\"', ';' }, CharsetStart);
        string RealCharset = 
               strWebPage.Substring(CharsetStart, CharsetEnd - CharsetStart);

        // real charset meta-tag in HTML differs from supplied server header???
        if(RealCharset!=Charset)
        {
            // get correct encoding
            Encoding CorrectEncoding = Encoding.GetEncoding(RealCharset);

            // read the web page again, but with correct encoding this time
            //   create request
            System.Net.WebRequest objRequest2 = System.Net.HttpWebRequest.Create(sURL);
            //   get response
            System.Net.HttpWebResponse objResponse2;
            objResponse2 = (System.Net.HttpWebResponse)objRequest2.GetResponse();
            //   read response
            using (StreamReader sr = 
              new StreamReader(objResponse2.GetResponseStream(), CorrectEncoding))
            {
                strWebPage = sr.ReadToEnd();
                // Close and clean up the StreamReader
                sr.Close();
            }
        }
    }

More Related Contents:

  1. Encode string URL without using HttpServerUtility [duplicate]
  2. Handling special characters [closed]
  3. How can I detect the encoding/codepage of a text file
  4. Convert a Unicode string to an escaped ASCII string
  5. Converting string to byte array in C#
  6. Determine a string’s encoding in C#
  7. How to achieve Base64 URL safe encoding in C#?
  8. Effective way to find any file’s Encoding
  9. How can I convert a hex string to a byte array? [duplicate]
  10. C# Convert string from UTF-8 to ISO-8859-1 (Latin1) H
  11. How to detect the character encoding of a text file?
  12. How can I transform string to UTF-8 in C#?
  13. C#: Class for decoding Quoted-Printable encoding?
  14. Strip the byte order mark from string in C#
  15. Does C# have an equivalent to JavaScript’s encodeURIComponent()?
  16. How do I decode a base64 encoded string?
  17. How do you remove invalid hexadecimal characters from an XML-based data source prior to constructing an XmlReader or XPathDocument that uses the data?
  18. How to get ASCII value of string in C#
  19. Signing and verifying signatures with RSA C#
  20. C# Help reading foreign characters using StreamReader
  21. Is there a standard way to encode a .NET string into JavaScript string for use in MS Ajax?
  22. How do I write out a text file in C# with a code page other than UTF-8?
  23. Encoding.GetEncoding can’t work in UWP app
  24. What’s the difference between UTF8/UTF16 and Base64 in terms of encoding
  25. Encoding parameters for a URL
  26. How to put an encoding attribute to xml other that utf-16 with XmlWriter?
  27. ASCIIEncoding In Windows Phone 7
  28. Convert UTF-8 to base64 string
  29. System.Net.Mail and =?utf-8?B?XXXXX…. Headers
  30. Cyrillic encoding in C#

Leave a Comment