How to parallelize for-loop in bash limiting number of processes

by

bash 4.4 will have an interesting new type of parameter expansion that simplifies Charles Duffy’s answer.

#!/bin/bash

num_procs=$1
num_iters=$2
num_jobs="\j"  # The prompt escape for number of jobs currently running
for ((i=0; i<num_iters; i++)); do
  while (( ${num_jobs@P} >= num_procs )); do
    wait -n
  done
  python foo.py "$i" arg2 &
done

More Related Contents:

  1. How do I iterate over a range of numbers defined by variables in Bash?
  2. Looping over pairs of values in bash [duplicate]
  3. How to loop over files in directory and change path and add suffix to filename
  4. How do you run multiple programs in parallel from a bash script?
  5. How to perform a for loop on each character in a string in Bash?
  6. How to skip the for loop when there are no matching files?
  7. Looping through alphabets in Bash
  8. Running a limited number of child processes in parallel in bash? [duplicate]
  9. How do I write a ‘for’ loop in Bash?
  10. How to add leading zeros for for-loop in shell? [duplicate]
  11. Is there a way to make bash job control quiet?
  12. Running programs in parallel using xargs
  13. Using a variable in brace expansion range fed to a for loop
  14. Running several scripts in parallel bash script [duplicate]
  15. Loop over tuples in bash?
  16. Parallel wget in Bash [duplicate]
  17. Expansion of variables inside single quotes in a command in Bash
  18. How can I write a heredoc to a file in Bash script?
  19. How to remove files starting with double hyphen?
  20. awk in bash with ls and variable
  21. How to loop through dates using Bash?
  22. Why does getopts only work the first time?
  23. How to pipe multiple commands into a single command in the shell? (sh, bash, …)
  24. find -exec with multiple commands
  25. How do I read user input into a variable in Bash?
  26. What is the best way to count “find” results?
  27. How to read from a file or standard input in Bash
  28. Count number of lines in a git repository
  29. Signals received by bash when terminal is closed
  30. Left side of pipe is the subshell?

Leave a Comment