Bash: limit the number of concurrent jobs? [duplicate]

by

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

parallel gzip ::: *.log

which will run one gzip per CPU core until all logfiles are gzipped.

If it is part of a larger loop you can use sem instead:

for i in *.log ; do
    echo $i Do more stuff here
    sem -j+0 gzip $i ";" echo done
done
sem --wait

It will do the same, but give you a chance to do more stuff for each file.

If GNU Parallel is not packaged for your distribution you can install GNU Parallel simply by:

$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
   fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh

It will download, check signature, and do a personal installation if it cannot install globally.

Watch the intro videos for GNU Parallel to learn more:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

More Related Contents:

  1. What does 2>&1 mean here?
  2. Input data to a shell script [closed]
  3. Why would a correct shell script give a wrapped/truncated/corrupted error message? [duplicate]
  4. When do we need curly braces around shell variables?
  5. Replace one substring for another string in shell script
  6. How to find the difference in days between two dates?
  7. How to assign the output of a Bash command to a variable? [duplicate]
  8. Capture stdout and stderr into different variables
  9. Checking from shell script if a directory contains files
  10. How to determine the current interactive shell that I’m in (command-line)
  11. Associative arrays in Shell scripts
  12. How to get the part of a file after the first line that matches a regular expression
  13. Trying to split a string into two variables
  14. Escape dollar sign in string by shell script
  15. Piping tail output though grep twice
  16. bash set -e and i=0;let i++ do not agree
  17. How do I echo stars (*) when reading password with `read`?
  18. Why is this bash prompt acting strangely/disappearing, and how do I fix it (OS X)?
  19. Bash: Copy named files recursively, preserving folder structure
  20. Read a variable in bash with a default value
  21. Counting lines or enumerating line numbers so I can loop over them – why is this an anti-pattern?
  22. Execute command containing quotes from shell variable [duplicate]
  23. What does “export” do in shell programming? [duplicate]
  24. What is the most elegant way to remove a path from the $PATH variable in Bash?
  25. Why equal to operator does not work if it is not surrounded by space?
  26. In bash, is there an equivalent of die “error msg”
  27. How to match a single quote in sed
  28. How do I integrate MSYS2 shell into Visual studio code on Window?
  29. Use GNU find to show only the leaf directories
  30. Get the name of the caller script in bash script

Leave a Comment