Bash `wait` command, waiting for more than 1 PID to finish execution

by

Let’s go through your options.

Wait for all background jobs, unconditionally

for i in 1 2 3 4 5; do
    cmd &
done
wait

This has the benefit of being simple, but you can’t keep your machine busy. If you want to start new jobs as old ones complete, you can’t. You machine gets less and less utilized until all the background jobs complete, at which point you can start a new batch of jobs.

Related is the ability to wait for a subset of jobs by passing multiple arguments to wait:

unrelated_job &
for i in 1 2 3 4 5; do
  cmd & pids+=($!)
done
wait "${pids[@]}"   # Does not wait for unrelated_job, though

Wait for individual jobs in arbitrary order

for i in 1 2 3 4 5; do
   cmd & pids+=($!)
done

for pid in "${pids[@]}"; do
   wait "$pid"
   # do something when a job completes
done

This has the benefit of letting you do work after a job completes, but
still has the problem that jobs other than $pid might complete first, leaving your machine underutilized until $pid actually completes. You do, however, still get the exit status for each individual job, even if it completes before you actually wait for it.

Wait for the next job to complete (bash 4.3 or later)

for i in 1 2 3 4 5; do
   cmd & pids+=($!)
done

for pid in "${pids[@]}"; do
   wait -n
   # do something when a job completes
done

Here, you can wait until a job completes, which means you can keep your machine as busy as possible. The only problem is, you don’t necessarily know which job completed, without using jobs to get the list of active processes and comparing it to pids.

Other options?

The shell by itself is not an ideal platform for doing job distribution, which is why there are a multitude of programs designed for managing batch jobs: xargs, parallel, slurm, qsub, etc.

More Related Contents:

  1. What is the best way to ensure only one instance of a Bash script is running? [duplicate]
  2. Why should eval be avoided in Bash, and what should I use instead?
  3. How to redirect output to a file and stdout
  4. ./configure : /bin/sh^M : bad interpreter [duplicate]
  5. The ‘eval’ command in Bash and its typical uses
  6. How to join multiple lines of file names into one with custom delimiter?
  7. Linux: copy and create destination dir if it does not exist
  8. How to redirect the output of the time command to a file in Linux?
  9. Linux PID recycling [closed]
  10. How to handle more than 10 parameters in shell
  11. How to count lines in a document? [closed]
  12. Changing default shell in Linux [closed]
  13. Linux Script to check if process is running and act on the result
  14. Example of using named pipes in Linux shell (Bash)
  15. Passing variables in remote ssh command
  16. Escape filenames the same way Bash does it
  17. Syntax error: “(” unexpected — with !(*.sh) in bash script
  18. How to execute a remote command over ssh with arguments?
  19. How can I find out a file’s MIME type (Content-Type)?
  20. How to exit if a command failed? [duplicate]
  21. How do I change bash history completion to complete what’s already on the line?
  22. How can I run a Windows executable from WSL (Ubuntu) Bash
  23. Using colors with printf
  24. Identify user in a Bash script called by sudo
  25. how to detect invalid utf8 unicode/binary in a text file
  26. Running shell script in parallel
  27. linux bash, camel case string to separate by dash
  28. File names with spaces in BASH
  29. What’s the point of eval/bash -c as opposed to just evaluating a variable?
  30. How to perform a for-each loop over all the files under a specified path?

Leave a Comment