How can you diff two pipelines in Bash?

by

A one-line with 2 tmp files (not what you want) would be:

 foo | bar > file1.txt && baz | quux > file2.txt && diff file1.txt file2.txt

With bash, you might try though:

 diff <(foo | bar) <(baz | quux)

 foo | bar | diff - <(baz | quux)  # or only use process substitution once

The 2nd version will more clearly remind you which input was which, by showing
-- /dev/stdin vs. ++ /dev/fd/63 or something, instead of two numbered fds.

Not even a named pipe will appear in the filesystem, at least on OSes where bash can implement process substitution by using filenames like /dev/fd/63 to get a filename that the command can open and read from to actually read from an already-open file descriptor that bash set up before exec’ing the command. (i.e. bash uses pipe(2) before fork, and then dup2 to redirect from the output of quux to an input file descriptor for diff, on fd 63.)

On a system with no “magical” /dev/fd or /proc/self/fd, bash might use named pipes to implement process substitution, but it would at least manage them itself, unlike temporary files, and your data wouldn’t be written to the filesystem.

You can check how bash implements process substitution with echo <(true) to print the filename instead of reading from it. It prints /dev/fd/63 on a typical Linux system. Or for more details on exactly what system calls bash uses, this command on a Linux system will trace file and file-descriptor system calls

strace -f -efile,desc,clone,execve bash -c '/bin/true | diff -u - <(/bin/true)'

Without bash, you could make a named pipe. Use - to tell diff to read one input from STDIN, and use the named pipe as the other:

mkfifo file1_pipe.txt
foo|bar > file1_pipe.txt && baz | quux | diff file1_pipe.txt - && rm file1_pipe.txt

Note that you can only pipe one output to multiple inputs with the tee command:

ls *.txt | tee /dev/tty txtlist.txt

The above command displays the output of ls *.txt to the terminal and outputs it to the text file txtlist.txt.

But with process substitution, you can use tee to feed the same data into multiple pipelines:

cat *.txt | tee >(foo | bar > result1.txt)  >(baz | quux > result2.txt) | foobar

More Related Contents:

  1. Fast way of finding lines in one file that are not in another?
  2. How do I tell if a regular file does not exist in Bash?
  3. How can I check if a program exists from a Bash script?
  4. How can I compare numbers in Bash?
  5. How to mount a host directory in a Docker container
  6. Why equal to operator does not work if it is surrounded by space?
  7. Set environment variables from file of key/value pairs
  8. Capturing Groups From a Grep RegEx
  9. Jq to replace text directly on file (like sed -i)
  10. Shell redirection i/o order
  11. Loop through all the files with a specific extension
  12. Colorized grep — viewing the entire file with highlighted matches
  13. How to pass in password to pg_dump?
  14. How can I profile a Bash shell script slow startup?
  15. Use Bash to read line by line and keep space
  16. How can you run a command in bash over and over until success?
  17. Minimal web server using netcat
  18. Hexadecimal To Decimal in Shell Script
  19. How to run two commands with sudo?
  20. bash zcat head causes pipefail?
  21. How to schedule to run first Sunday of every month
  22. Passing a variable to a remote host in a bash script with ssh and EOF [duplicate]
  23. Using cut command to remove multiple columns
  24. Looping over arrays, printing both index and value
  25. Temporarily change current working directory in bash to run a command [duplicate]
  26. Generating random number between 1 and 10 in Bash Shell Script [duplicate]
  27. How to break out of a loop in Bash?
  28. How to assign a glob expression to a variable in a Bash script?
  29. Access arguments to Bash script inside a function [duplicate]
  30. What’s the difference between “here string” and echo + pipe

Leave a Comment