Reading input files by line using read command in shell scripting skips last line

by

read reads until it finds a newline character or the end of file, and returns a non-zero exit code if it encounters an end-of-file. So it’s quite possible for it to both read a line and return a non-zero exit code.

Consequently, the following code is not safe if the input might not be terminated by a newline:

while read LINE; do
  # do something with LINE
done

because the body of the while won’t be executed on the last line.

Technically speaking, a file not terminated with a newline is not a text file, and text tools may fail in odd ways on such a file. However, I’m always reluctant to fall back on that explanation.

One way to solve the problem is to test if what was read is non-empty (-n):

while read -r LINE || [[ -n $LINE ]]; do
  # do something with LINE
done

Other solutions include using mapfile to read the file into an array, piping the file through some utility which is guaranteed to terminate the last line properly (grep ., for example, if you don’t want to deal with blank lines), or doing the iterative processing with a tool like awk (which is usually my preference).

Note that -r is almost certainly needed in the read builtin; it causes read to not reinterpret \-sequences in the input.

More Related Contents:

  1. Parse HTML using shell
  2. How do I tell if a regular file does not exist in Bash?
  3. While loop stops reading after the first line in Bash
  4. How can I compare numbers in Bash?
  5. Pipe output and capture exit status in Bash
  6. Redirect stderr and stdout in Bash [duplicate]
  7. Why equal to operator does not work if it is surrounded by space?
  8. Which characters need to be escaped when using Bash?
  9. Capturing Groups From a Grep RegEx
  10. Why does “local” sweep the return code of a command?
  11. bash : Illegal number
  12. Batch renaming files with Bash
  13. Shell redirection i/o order
  14. How to generate random number in Bash?
  15. In bash, how do I bind a function key to a command?
  16. “Invalid Arithmetic Operator” when doing floating-point math in bash
  17. Using sed to mass rename files
  18. Colorized grep — viewing the entire file with highlighted matches
  19. How to pass in password to pg_dump?
  20. Select random lines from a file
  21. Variables getting reset after the while read loop that reads from a pipeline
  22. Hexadecimal To Decimal in Shell Script
  23. Should I use a Shebang with Bash scripts?
  24. Bash scripting, multiple conditions in while loop
  25. How to schedule to run first Sunday of every month
  26. Printing asterisk (“*”) in bash shell
  27. Why would I create an alias which creates a function?
  28. Rename all files in directory from $filename_h to $filename_half?
  29. Temporarily change current working directory in bash to run a command [duplicate]
  30. BASH: Writing a Script to Recursively Travel a Directory of N Levels

Leave a Comment