Using awk to remove the Byte-order mark

by

Using GNU sed (on Linux or Cygwin):

# Removing BOM from all text files in current directory:
sed -i '1 s/^\xef\xbb\xbf//' *.txt

On FreeBSD:

sed -i .bak '1 s/^\xef\xbb\xbf//' *.txt

Advantage of using GNU or FreeBSD sed: the -i parameter means “in place”, and will update files without the need for redirections or weird tricks.

On Mac:

This awk solution in another answer works, but the sed command above does not work. At least on Mac (Sierra) sed documentation does not mention supporting hexadecimal escaping ala \xef.

A similar trick can be achieved with any program by piping to the sponge tool from moreutils:

awk '…' INFILE | sponge INFILE

More Related Contents:

  1. What’s the difference between UTF-8 and UTF-8 without BOM?
  2. What’s the difference between UTF-8 and UTF-8 with BOM?
  3. How to use unicode characters in Windows command line?
  4. Unicode, UTF, ASCII, ANSI format differences
  5. Is there a Windows command shell that will display Unicode characters?
  6. What’s the difference between ASCII and Unicode?
  7. Encoding.UTF8.GetString doesn’t take into account the Preamble/BOM
  8. Read a UTF-8 text file with BOM
  9. What’s the best option to display Unicode text (hebrew, etc.) in VB6
  10. Unicode range for Japanese
  11. Using redirection within the script produces a unicode output. How to emit single-byte ASCII text?
  12. How can I clean source code files of invisible characters?
  13. In Unicode, why are there two representations for the Arabic digits?
  14. Can UTF-8 contain zero byte?
  15. List of all unicode’s open/close brackets?
  16. grepping binary files and UTF16
  17. How to avoid tripping over UTF-8 BOM when reading files
  18. Printing Unicode characters to the PowerShell prompt
  19. Difference between Big Endian and little Endian Byte order
  20. iconv: Converting from Windows ANSI to UTF-8 with BOM
  21. Print Directory & File Structure with icons for representation in Markdown [closed]
  22. How to read utf16 text file to string in golang?
  23. Classic ASP: How to write unicode string data in classic ASP?
  24. What does it mean when my text is displayed as Question Marks?
  25. Convert unicode code point to literal character in Go
  26. Should I support Unicode in passwords?
  27. What does u’\ufe0f’ in an emoji mean? Is it the same if I delete it?
  28. UTF-8 Continuation bytes
  29. How can I use Unicode characters on the Windows command line?
  30. Why is Unicode restricted to 0x10FFFF?

Leave a Comment