how to detect invalid utf8 unicode/binary in a text file

by

Assuming you have your locale set to UTF-8 (see locale output), this works well to recognize invalid UTF-8 sequences:

grep -axv '.*' file.txt

Explanation (from grep man page):

  • -a, –text: treats file as text, essential prevents grep to abort once finding an invalid byte sequence (not being utf8)
  • -v, –invert-match: inverts the output showing lines not matched
  • -x ‘.*’ (–line-regexp): means to match a complete line consisting of any utf8 character.

Hence, there will be output, which is the lines containing the invalid not utf8 byte sequence containing lines (since inverted -v)

More Related Contents:

  1. How to remove non UTF-8 characters from text file
  2. grep lines that contain 1 character followed by another character
  3. Bash script prints “Command Not Found” on empty lines
  4. What’s the magic of “-” (a dash) in command-line parameters?
  5. Pseudo-terminal will not be allocated because stdin is not a terminal
  6. Is there a “goto” statement in bash?
  7. Expanding a bash array only gives the first element
  8. How can I generate new variable names on the fly in a shell script?
  9. How can I preserve quotes in printing a bash script’s arguments
  10. Bash while read loop extremely slow compared to cat, why?
  11. Automatically capture output of last command into a variable using Bash?
  12. Find multiple files and rename them in Linux
  13. How to find the last field using ‘cut’
  14. How can I loop over the output of a shell command?
  15. Difference between ${} and $() in Bash [duplicate]
  16. Why does /bin/sh behave differently to /bin/bash even if one points to the other?
  17. [ :Unexpected operator in shell programming [duplicate]
  18. How to redirect output of an already running process [duplicate]
  19. Command to get time in milliseconds
  20. How to create a hex dump of file containing only the hex characters without spaces in bash?
  21. How to sort strings that contain a common prefix and suffix numerically from Bash?
  22. How to create a CPU spike with a bash command
  23. linux: kill background task
  24. Why can’t I use Unix Nohup with Bash For-loop?
  25. How can I set an environment variable only for the duration of the script?
  26. Linux/Unix command to determine if process is running?
  27. Rename files and directories recursively under ubuntu /bash
  28. Get ceiling integer from number in linux (BASH)
  29. How can I trim white space from a variable in awk?
  30. Maximum number of Bash arguments != max num cp arguments?

Leave a Comment