Since you state: All of my (text) files are currently stored in UTF-8 with the BOM then use the ‘utf-8-sig’ codec to decode them: >>> s = u’Hello, world!’.encode(‘utf-8-sig’) >>> s ‘\xef\xbb\xbfHello, world!’ >>> s.decode(‘utf-8-sig’) u’Hello, world!’ It automatically removes the expected BOM, and works correctly if the BOM is not present as well.
Java does not handle BOM properly. In fact Java handles a BOM like every other char. Found this: http://www.rgagnon.com/javadetails/java-handle-utf8-file-with-bom.html public static final String UTF8_BOM = “\uFEFF”; private static String removeUTF8BOM(String s) { if (s.startsWith(UTF8_BOM)) { s = s.substring(1); } return s; } May be I would use apache IO instead: http://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/input/BOMInputStream.html
Your point about UTF-8 not being able to store all characters you need is invalid. UTF-8 is able to store every character defined in the Unicode standard. The only difference is that, for text in certain languages, UTF-8 can take more space to store its codepoints than, say, UTF-16. The opposite is also true: for … Read more
With ruby >= 1.9.2 you can use the mode r:bom|utf-8 This should work (I haven’t test it in combination with json): json = nil #define the variable outside the block to keep the data File.open(‘file.txt’, “r:bom|utf-8”){|file| json = JSON.parse(file.read) } It doesn’t matter, if the BOM is available in the file or not. Andrew remarked, … Read more
Well it writes the BOM because you are instructing it to, in the line Encoding utf8WithoutBom = new UTF8Encoding(true); true means that the BOM should be emitted, using Encoding utf8WithoutBom = new UTF8Encoding(encoderShouldEmitUTF8Identifier: false); writes no BOM. My objective is create a file using UTF-8 as Encoding and 8859-1 as CharSet Sadly, this is not … Read more
Using VIM Open file in VIM: vi text.xml Remove BOM encoding: :set nobomb Save and quit: :wq For a non-interactive solution, try the following command line: vi -c “:set nobomb” -c “:wq” text.xml That should remove the BOM, save the file and quit, all from the command line.
Prepend \ufeff to the string. See http://msdn.microsoft.com/en-us/library/ie/2yfce773(v=vs.94).aspx See discussion between @jeff-fischer and @casey for details on UTF-8 and UTF-16 and the BOM. What actually makes the above work is that the string \ufeff is always used to represent the BOM, regardless of UTF-8 or UTF-16 being used. See p.36 in The Unicode Standard 5.0, Chapter … Read more
With ruby 1.9.2 you can use the mode r:bom|utf-8 text_without_bom = nil #define the variable outside the block to keep the data File.open(‘file.txt’, “r:bom|utf-8”){|file| text_without_bom = file.read } or text_without_bom = File.read(‘file.txt’, encoding: ‘bom|utf-8’) or text_without_bom = File.read(‘file.txt’, mode: ‘r:bom|utf-8′) It doesn’t matter, if the BOM is available in the file or not. You may … Read more
The UTF-8 BOM is a sequence of bytes at the start of a text stream (0xEF, 0xBB, 0xBF) that allows the reader to more reliably guess a file as being encoded in UTF-8. Normally, the BOM is used to signal the endianness of an encoding, but since endianness is irrelevant to UTF-8, the BOM is … Read more
You’ve got a Unicode UTF-8 BOM at the start of the file: http://en.wikipedia.org/wiki/Byte_order_mark A text editor or web browser interpreting the text as ISO-8859-1 or CP1252 will display the characters  for this R is giving you the ï and then converting the other two into dots as they are non-alphanumeric characters. Here: http://r.789695.n4.nabble.com/Writing-Unicode-Text-into-Text-File-from-R-in-Windows-td4684693.html Duncan … Read more