Effective way to find any file’s Encoding

by

The StreamReader.CurrentEncoding property rarely returns the correct text file encoding for me. I’ve had greater success determining a file’s endianness, by analyzing its byte order mark (BOM). If the file does not have a BOM, this cannot determine the file’s encoding.

*UPDATED 4/08/2020 to include UTF-32LE detection and return correct encoding for UTF-32BE

/// <summary>
/// Determines a text file's encoding by analyzing its byte order mark (BOM).
/// Defaults to ASCII when detection of the text file's endianness fails.
/// </summary>
/// <param name="filename">The text file to analyze.</param>
/// <returns>The detected encoding.</returns>
public static Encoding GetEncoding(string filename)
{
    // Read the BOM
    var bom = new byte[4];
    using (var file = new FileStream(filename, FileMode.Open, FileAccess.Read))
    {
        file.Read(bom, 0, 4);
    }

    // Analyze the BOM
    if (bom[0] == 0x2b && bom[1] == 0x2f && bom[2] == 0x76) return Encoding.UTF7;
    if (bom[0] == 0xef && bom[1] == 0xbb && bom[2] == 0xbf) return Encoding.UTF8;
    if (bom[0] == 0xff && bom[1] == 0xfe && bom[2] == 0 && bom[3] == 0) return Encoding.UTF32; //UTF-32LE
    if (bom[0] == 0xff && bom[1] == 0xfe) return Encoding.Unicode; //UTF-16LE
    if (bom[0] == 0xfe && bom[1] == 0xff) return Encoding.BigEndianUnicode; //UTF-16BE
    if (bom[0] == 0 && bom[1] == 0 && bom[2] == 0xfe && bom[3] == 0xff) return new UTF32Encoding(true, true);  //UTF-32BE

    // We actually have no idea what the encoding is if we reach this point, so
    // you may wish to return null instead of defaulting to ASCII
    return Encoding.ASCII;
}

More Related Contents:

  1. Encode string URL without using HttpServerUtility [duplicate]
  2. Handling special characters [closed]
  3. How can I detect the encoding/codepage of a text file
  4. Convert a Unicode string to an escaped ASCII string
  5. Converting string to byte array in C#
  6. Determine a string’s encoding in C#
  7. How to achieve Base64 URL safe encoding in C#?
  8. How can I convert a hex string to a byte array? [duplicate]
  9. C# Convert string from UTF-8 to ISO-8859-1 (Latin1) H
  10. How to detect the character encoding of a text file?
  11. How can I transform string to UTF-8 in C#?
  12. C#: Class for decoding Quoted-Printable encoding?
  13. Strip the byte order mark from string in C#
  14. Does C# have an equivalent to JavaScript’s encodeURIComponent()?
  15. How do I decode a base64 encoded string?
  16. How do you remove invalid hexadecimal characters from an XML-based data source prior to constructing an XmlReader or XPathDocument that uses the data?
  17. How to get ASCII value of string in C#
  18. Signing and verifying signatures with RSA C#
  19. C# Help reading foreign characters using StreamReader
  20. Is there a standard way to encode a .NET string into JavaScript string for use in MS Ajax?
  21. Encoding trouble with HttpWebResponse
  22. How do I write out a text file in C# with a code page other than UTF-8?
  23. Encoding.GetEncoding can’t work in UWP app
  24. What’s the difference between UTF8/UTF16 and Base64 in terms of encoding
  25. Encoding parameters for a URL
  26. How to put an encoding attribute to xml other that utf-16 with XmlWriter?
  27. ASCIIEncoding In Windows Phone 7
  28. Convert UTF-8 to base64 string
  29. System.Net.Mail and =?utf-8?B?XXXXX…. Headers
  30. Cyrillic encoding in C#

Leave a Comment