How to identify the file content as ASCII or binary

by

If a file contains only the decimal bytes 9–13, 32–126, it’s probably a pure ASCII text file. Otherwise, it’s not. However, it may still be text in another encoding.

If, in addition to the above bytes, the file contains only the decimal bytes 128–255, it’s probably a text file in an 8-bit or variable-length ASCII-based encoding such as ISO-8859-1, UTF-8 or ASCII+Big5. If not, for some purposes you may be able to stop here and consider the file to be binary. However, it may still be text in a 16- or 32-bit encoding.

If a file doesn’t meet the above constraints, examine the first 2–4 bytes of the file for a byte-order mark:

  • If the first two bytes are hex FE FF, the file is tentatively UTF-16 BE.
  • If the first two bytes are hex FF FE, and the following two bytes are not hex 00 00 , the file is tentatively UTF-16 LE.
  • If the first four bytes are hex 00 00 FE FF, the file is tentatively UTF-32 BE.
  • If the first four bytes are hex FF FE 00 00, the file is tentatively UTF-32 LE.

If, through the above checks, you have determined a tentative encoding, then check only for the corresponding encoding below, to ensure that the file is not a binary file which happens to match a byte-order mark.

If you have not determined a tentative encoding, the file might still be a text file in one of these encodings, since the byte-order mark is not mandatory, so check for all encodings in the following list:

  • If the file contains only big-endian two-byte words with the decimal values 9–13, 32–126, and 128 or above, the file is probably UTF-16 BE.
  • If the file contains only little-endian two-byte words with the decimal values 9–13, 32–126, and 128 or above, the file is probably UTF-16 LE.
  • If the file contains only big-endian four-byte words with the decimal values 9–13, 32–126, and 128 or above, the file is probably UTF-32 BE.
  • If the file contains only little-endian four-byte words with the decimal values 9–13, 32–126, and 128 or above, the file is probably UTF-32 LE.

If, after all these checks, you still haven’t determined an encoding, the file isn’t a text file in any ASCII-based encoding I know about, so for most purposes you can probably consider it to be binary (it might still be a text file in a non-ASCII encoding such as EBCDIC, but I suspect that’s well outside the scope of your concern).

More Related Contents:

  1. How to shift characters to ASCII values in a file based on user input c++ [closed]
  2. Serializing a class which contains a std::string
  3. What is the idea behind ^= 32, that converts lowercase letters to upper and vice versa?
  4. Convert an int to ASCII character
  5. How to read a binary file into a vector of unsigned chars
  6. How to read / write a struct in Binary Files?
  7. Does C and C++ guarantee the ASCII of [a-f] and [A-F] characters?
  8. c++ Read/Write class from/to binary file
  9. Are the character digits [‘0’..’9′] required to have contiguous numeric values?
  10. How to convert a string of hex values to a string?
  11. are “seekp” & “seekg” interchangeable?
  12. Compilation error: expected constructor, destructor, or type conversion before ‘;’ token [closed]
  13. Difference between ‘struct’ and ‘typedef struct’ in C++?
  14. C/C++ maximum stack size of program on mainstream OSes
  15. A most vexing parse error: constructor with no arguments
  16. Convert CString to const char*
  17. How can I easily format my data table in C++?
  18. How do I set, clear, and toggle a single bit?
  19. Two calls to destructor
  20. Which Boost libraries are header-only?
  21. int to unsigned int conversion
  22. Can we see the template instantiated code by C++ compiler
  23. Pros and cons of using nested C++ classes and enumerations?
  24. C++/C function pointers that return void*
  25. C++11 operator”” with double parameter
  26. Some clarification needed about synchronous versus asynchronous asio operations
  27. error for hash function of pair of ints
  28. Is it possible to use threads to speed up file reading?
  29. “APIENTRY _tWinMain” and “WINAPI WinMain” difference
  30. Reading a Matrix txt file and storing as an array

Leave a Comment