Running shell script in parallel

by

Another very handy way to do this is with gnu parallel, which is well worth installing if you don’t already have it; this is invaluable if the tasks don’t necessarily take the same amount of time. 

seq 1000 | parallel -j 8 --workdir $PWD ./myrun {}

will launch ./myrun 1, ./myrun 2, etc, making sure 8 jobs at a time are running. It can also take lists of nodes if you want to run on several nodes at once, eg in a PBS job; our instructions to our users for how to do that on our system are here.

Updated to add: You want to make sure you’re using gnu-parallel, not the more limited utility of the same name that comes in the moreutils package (the divergent history of the two is described here.)

More Related Contents:

  1. When should I wrap quotes around a shell variable?
  2. What’s the magic of “-” (a dash) in command-line parameters?
  3. Aborting a shell script if any command returns a non-zero value
  4. What does $@ mean in a shell script?
  5. Better way to rename files based on multiple patterns
  6. Forcing bash to expand variables in a string loaded from a file
  7. Linux: copy and create destination dir if it does not exist
  8. What is the difference between using `sh` and `source`?
  9. Setting environment variables in Linux using Bash
  10. What is the difference between “#!/usr/bin/env bash” and “#!/usr/bin/bash”?
  11. Does “argument list too long” restriction apply to shell builtins?
  12. Given two directory trees, how can I find out which files differ by content? [closed]
  13. How to sort a file in-place?
  14. How to run a shell script on a Unix console or Mac terminal?
  15. Using the passwd command from within a shell script
  16. Is it OK to use the same input file as output of a piped command?
  17. When to wrap quotes around a shell variable?
  18. How do you normalize a file path in Bash?
  19. Linux/Unix command to determine if process is running?
  20. Difference between shell and environment variables
  21. What’s the point of eval/bash -c as opposed to just evaluating a variable?
  22. Need to remove the count from the output when using “uniq -c” command
  23. How do you run a script on login in *nix?
  24. create a file with prefix =file1 followed by the random number
  25. Why to code a script which will check if some package is installed in linux and if not then install it?
  26. Is there a way to change the environment variables of another process in Unix?
  27. Shell script: Run function from script over ssh
  28. How do I syntax check a Bash script without running it?
  29. How to attach a file using mail command on Linux? [duplicate]
  30. How to avoid race condition when using a lock-file to avoid two instances of a script running simultaneously?

Leave a Comment