You have to tell open that this is UTF-8 with BOM. I know that works with io.open: import io . . . inputFile = io.open(“test.csv”, “r”, encoding=’utf-8-sig’) . . . And you have to open the file in text mode, “r” instead of “rb”.
Maybe you can shell out to a Python script that uses Chardet: Universal Encoding Detector. It is a reimplementation of the character encoding detection that used by Firefox, and is used by many different applications. Useful links: Mozilla’s code, research paper it was based on (ironically, my Firefox fails to correctly detect the encoding of … Read more
If you look in the same menu. Click “Convert to UTF-8.”
Use an XmlTextWriter and pass that to the XDocument’s Save() method, that way you can have more control over the type of encoding used: var doc = new XDocument( new XDeclaration(“1.0”, “utf-8”, null), new XElement(“root”, new XAttribute(“note”, “boogers”)) ); using (var writer = new XmlTextWriter(“.\\boogers.xml”, new UTF8Encoding(false))) { doc.Save(writer); } The UTF8Encoding class constructor has … Read more
Have you tried read.csv(…, fileEncoding = “UTF-8-BOM”)?. ?file says: As from R 3.0.0 the encoding ‘”UTF-8-BOM”’ is accepted and will remove a Byte Order Mark if present (which it often is for files and webpages generated by Microsoft applications).
# vim file.xml :set nobomb :wq
Try like this: public ActionResult Download() { var data = Encoding.UTF8.GetBytes(“some data”); var result = Encoding.UTF8.GetPreamble().Concat(data).ToArray(); return File(result, “application/csv”, “foo.csv”); } The reason is that the UTF8Encoding constructor that takes a boolean parameter doesn’t do what you would expect: byte[] bytes = new UTF8Encoding(true).GetBytes(“a”); The resulting array would contain a single byte with the value … Read more
In order to omit the byte order mark (BOM), your stream must use an instance of UTF8Encoding other than System.Text.Encoding.UTF8 (which is configured to generate a BOM). There are two easy ways to do this: 1. Explicitly specifying a suitable encoding: Call the UTF8Encoding constructor with False for the encoderShouldEmitUTF8Identifier parameter. Pass the UTF8Encoding instance … Read more
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(…), StandardCharsets.UTF_8)); out.write(‘\ufeff’); out.write(…); This correctly writes out 0xEF 0xBB 0xBF to the file, which is the UTF-8 representation of the BOM.
It looks like this method ignores the BOM (Byte Order Mark), which might be a part of a legitimate binary representation of a UTF8 string, and takes it as a character. It doesn’t look like it “ignores” it at all – it faithfully converts it to the BOM character. That’s what it is, after all. … Read more