I believe the problem is that codecs.BOM_UTF8 is a byte string, not a Unicode string. I suspect the file handler is trying to guess what you really mean based on “I’m meant to be writing Unicode as UTF-8-encoded text, but you’ve given me a byte string!” Try writing the Unicode string for the byte order … Read more
It turns out, the answer is that what I’m seeing is a Byte Order Mark, which is a character that tells whatever is loading the document what it is encoded in. In my case, it’s encoded in utf-8, so the corresponding BOM was EF BB BF, as shown below. To remove it, I opened it … Read more
Indeed, the BOM is actual data sent to the browser. The browser will happily ignore it, but still you cannot send headers then. I believe the problem really is your and your friend’s editor settings. Without a BOM, your friend’s editor may not automatically recognize the file as UTF-8. He can try to set up … Read more
Simply use the “utf-8-sig” codec: fp = open(“file.txt”) s = fp.read() u = s.decode(“utf-8-sig”) That gives you a unicode string without the BOM. You can then use s = u.encode(“utf-8”) to get a normal UTF-8 encoded string back in s. If your files are big, then you should avoid reading them all into memory. The … Read more
You can’t depend on the file having a BOM. UTF-8 doesn’t require it. And non-Unicode encodings don’t even have a BOM. There are, however, other ways to detect the encoding. UTF-32 BOM is 00 00 FE FF (for BE) or FF FE 00 00 (for LE). But UTF-32 is easy to detect even without a … Read more
Using GNU sed (on Linux or Cygwin): # Removing BOM from all text files in current directory: sed -i ‘1 s/^\xef\xbb\xbf//’ *.txt On FreeBSD: sed -i .bak ‘1 s/^\xef\xbb\xbf//’ *.txt Advantage of using GNU or FreeBSD sed: the -i parameter means “in place”, and will update files without the need for redirections or weird tricks. … Read more
BOM or Byte Order Mark is sometimes quite annoying. Visual Studio does not change the file unless you save it (as Hans said). And here is the solution to your problem: If you want to save a file with other encodings, select save as and extend the save button in file dialog and select “Save … Read more
you would use the following code to remove utf8 bom //Remove UTF8 Bom function remove_utf8_bom($text) { $bom = pack(‘H*’,’EFBBBF’); $text = preg_replace(“/^$bom/”, ”, $text); return $text; }
Three words for you: Byte Order Mark (BOM) That’s the representation for the UTF-8 BOM in ISO-8859-1. You have to tell your editor to not use BOMs or use a different editor to strip them out. To automatize the BOM’s removal you can use awk as shown in this question. As another answer says, the … Read more
I have the same (or similar) problem. In my case, if I add a BOM to the output, it works: header(‘Content-Encoding: UTF-8’); header(‘Content-type: text/csv; charset=UTF-8’); header(‘Content-Disposition: attachment; filename=Customers_Export.csv’); echo “\xEF\xBB\xBF”; // UTF-8 BOM I believe this is a pretty ugly hack, but it worked for me, at least for Excel 2007 Windows. Not sure it’ll … Read more