Preserving leading white space while reading>>writing a file line by line in bash

by

As others have pointed out, using cat or awk instead of a read-echo loop is a much better way to do this — avoids the whitespace-trimming problem (and a couple of others you haven’t stumbled upon), runs faster, and at least with cat, is simply cleaner code. Nonetheless, I’d like to take a stab at getting the read-echo loop to work right.

First, the whitespace-trimming problem: the read command automatically trims leading and trailing whitespace; this can be fixed by changing its definition of whitespace by setting the IFS variable to blank. Also, read assumes that a backslash at the end of the line means the next line is a continuation, and should be spliced together with this one; to fix this, use its -r (raw) flag. The third problem here is that many implementations of echo interpret escape sequences in the string (e.g. they may turn \n into an actual newline); to fix this, use printf instead. Finally, just as a general scripting hygiene rule, you shouldn’t use cat when you don’t actually need to; use input redirection instead. With those changes, the inner loop looks like this:

while IFS='' read -r line; do 
  printf "%s\n" "$line">>$OUTPUT
done <$f

…there are also a couple of other problems with the surrounding script: the line that tries to define FILES as the list of available .textile files has quotes around it, meaning it never gets expanded into an actual list of files. The best way to do this is to use an array:

FILES=(../best-practices/*.textile)
...
for f in "${FILES[@]}"

(and all occurrences of $f should be in double-quotes in case any of the filenames have spaces or other funny characters in them — should really do this with $OUTPUT as well, though since that’s defined in the script it’s actually safe to leave off.)

Finally, there’s a echo "">$OUTPUT near the top of the loop-over-files that’s going to erase the output file every time through (i.e. at the end, it only contains the last .textile file); this needs to be moved to before the loop. I’m not sure if the intent here was to put a single blank line at the beginning of the file, or three blank lines between files (and one at the beginning and two at the end), so I’m not sure exactly what the appropriate replacement is. Anyway, here’s what I can up with after fixing all of these problems:

#!/bin/sh
OUTPUT="../best_practices.textile"
FILES=(../best-practices/*.textile)

: >"$OUTPUT"
for f in "${FILES[@]}"
do
  echo "Processing $f file..."
  echo >>"$OUTPUT"

  while IFS='' read -r line; do 
    printf "%s\n" "$line">>"$OUTPUT"
  done <"$f"

  echo >>"$OUTPUT"
  echo >>"$OUTPUT"
done

More Related Contents:

  1. An efficient way to transpose a file in Bash
  2. How to address error “bash: !d’: event not found” in Bash command substitution [duplicate]
  3. Remove Unicode characters from textfiles – sed , other Bash/shell methods
  4. Parsing variables from config file in Bash
  5. Parse Date in Bash
  6. Reading input files by line using read command in shell scripting skips last line
  7. Parse HTML using shell
  8. Echo newline in Bash prints literal \n
  9. In the shell, what does ” 2>&1 ” mean?
  10. ssh breaks out of while-loop in bash [duplicate]
  11. Remove a fixed prefix/suffix from a string in Bash
  12. I just assigned a variable, but echo $variable shows something else
  13. Bash how do you capture stderr to a variable? [duplicate]
  14. How to use aliases defined in .bashrc in other scripts?
  15. Should aliases go in .bashrc or .bash_profile? [duplicate]
  16. How to read user input into a variable in Bash?
  17. How to provide password to a command that prompts for one in bash?
  18. Deleting lines from one file which are in another file
  19. source command not found in sh shell
  20. How to check if a file contains a specific string using Bash
  21. How to enter ssh password using bash? [duplicate]
  22. What is a unix command for deleting the first N characters of a line?
  23. How do I escape slashes and double and single quotes in sed?
  24. How to suppress variable substitution in bash heredocs
  25. Error message on Terminal launch [duplicate]
  26. Exit when one process in pipe fails
  27. How to split a file into equal parts, without breaking individual lines? [duplicate]
  28. Prevent grep returning an error when input doesn’t match
  29. Read stdin for user input when a file is already redirected to stdin [duplicate]
  30. What is the shortest way to get n-th column of an output?

Leave a Comment