Why piping input to “read” only works when fed into “while read …” construct? [duplicate]

by

How to do a loop against stdin and get result stored in a variable

Under bash (and other shell also), when you pipe something to another command via |, you will implicitly create a fork, a subshell that is a child of current session. The subshell can’t affect current session’s environment.

So this:

TOTAL=0
printf "%s %s\n" 9 4 3 1 77 2 25 12 226 664 |
  while read A B;do
      ((TOTAL+=A-B))
      printf "%3d - %3d = %4d -> TOTAL= %4d\n" $A $B $[A-B] $TOTAL
    done
echo final total: $TOTAL

won’t give expected result! :

  9 -   4 =    5 -> TOTAL=    5
  3 -   1 =    2 -> TOTAL=    7
 77 -   2 =   75 -> TOTAL=   82
 25 -  12 =   13 -> TOTAL=   95
226 - 664 = -438 -> TOTAL= -343
echo final total: $TOTAL
final total: 0

Where computed TOTAL could’nt be reused in main script.

Inverting the fork

By using bash Process Substitution, Here Documents or Here Strings, you could inverse the fork:

Here strings

read A B <<<"first second"
echo $A
first

echo $B
second

Here Documents

while read A B;do
    echo $A-$B
    C=$A-$B
  done << eodoc
first second
third fourth
eodoc
first-second
third-fourth

outside of the loop:

echo : $C
: third-fourth

Here Commands

TOTAL=0
while read A B;do
    ((TOTAL+=A-B))
    printf "%3d - %3d = %4d -> TOTAL= %4d\n" $A $B $[A-B] $TOTAL
  done < <(
    printf "%s %s\n" 9 4 3 1 77 2 25 12 226 664
)
  9 -   4 =    5 -> TOTAL=    5
  3 -   1 =    2 -> TOTAL=    7
 77 -   2 =   75 -> TOTAL=   82
 25 -  12 =   13 -> TOTAL=   95
226 - 664 = -438 -> TOTAL= -343

# and finally out of loop:
echo $TOTAL
-343

Now you could use $TOTAL in your main script.

Piping to a command list

But for working only against stdin, you may create a kind of script into the fork:

printf "%s %s\n" 9 4 3 1 77 2 25 12 226 664 | {
    TOTAL=0
    while read A B;do
        ((TOTAL+=A-B))
        printf "%3d - %3d = %4d -> TOTAL= %4d\n" $A $B $[A-B] $TOTAL
    done
    echo "Out of the loop total:" $TOTAL
  }

Will give:

  9 -   4 =    5 -> TOTAL=    5
  3 -   1 =    2 -> TOTAL=    7
 77 -   2 =   75 -> TOTAL=   82
 25 -  12 =   13 -> TOTAL=   95
226 - 664 = -438 -> TOTAL= -343
Out of the loop total: -343

Note: $TOTAL could not be used in main script (after last right curly bracket } ).

Using lastpipe bash option

As @CharlesDuffy correctly pointed out, there is a bash option used to change this behaviour. But for this, we have to first disable job control:

shopt -s lastpipe           # Set *lastpipe* option
set +m                      # Disabling job control
TOTAL=0
printf "%s %s\n" 9 4 3 1 77 2 25 12 226 664 |
  while read A B;do
      ((TOTAL+=A-B))
      printf "%3d - %3d = %4d -> TOTAL= %4d\n" $A $B $[A-B] $TOTAL
    done

  9 -   4 =    5 -> TOTAL= -338
  3 -   1 =    2 -> TOTAL= -336
 77 -   2 =   75 -> TOTAL= -261
 25 -  12 =   13 -> TOTAL= -248
226 - 664 = -438 -> TOTAL= -686

echo final total: $TOTAL
-343

This will work, but I (personally) don’t like this because this is not standard and won’t help to make script readable. Also disabling job control seem expensive for accessing this behaviour.

Note: Job control is enabled by default only in interactive sessions. So set +m is not required in normal scripts.

So forgotten set +m in a script would create different behaviours if run in a console or if run in a script. This will not going to make this easy to understand or to debug…

More Related Contents:

  1. How to pipe input to a Bash while loop and preserve variables after loop ends
  2. A variable modified inside a while loop is not remembered
  3. While loop stops reading after the first line in Bash
  4. How can I pipe stderr, and not stdout?
  5. Pipe output and capture exit status in Bash
  6. How to trick an application into thinking its stdout is a terminal, not a pipe
  7. What is a simple explanation for how pipes work in Bash?
  8. How can I detect if my shell script is running through a pipe?
  9. Piping both stdout and stderr in bash?
  10. Variables getting reset after the while read loop that reads from a pipeline
  11. How can you run a command in bash over and over until success?
  12. Reading key/value parameters from a file into a shell script
  13. Capture both stdout and stderr in Bash [duplicate]
  14. Piping and Redirection
  15. How to get the PID of a process that is piped to another process in Bash?
  16. How to pipe multiple commands into a single command in the shell? (sh, bash, …)
  17. Pipe, standard input and command line arguments in Bash
  18. How to pipe stdout while keeping it on screen ? (and not to a output file)
  19. pipe stdout and stderr to two different processes in shell script?
  20. Bash scripting, multiple conditions in while loop
  21. Syntax for a single-line while loop in Bash
  22. Pipe output to two different commands [duplicate]
  23. Looping through find output in Bash where file name contains white spaces
  24. Get exit code from subshell through the pipes
  25. bash piping prevents global variable assignment
  26. while loop to read file ends prematurely
  27. How to break out of a loop in Bash?
  28. How to detect if a Node.js script is running through a shell pipe?
  29. What’s the difference between “here string” and echo + pipe
  30. Left side of pipe is the subshell?

Leave a Comment