Running programs in parallel using xargs

by

From the xargs man page:

This manual page documents the GNU version of xargs. xargs reads items
from the standard input, delimited by blanks (which can be protected
with double or single quotes or a backslash) or newlines, and executes
the command (default is /bin/echo) one or more times with any initial-
arguments followed by items read from standard input. Blank lines on
the standard input are ignored.

Which means that for your example xargs is waiting and collecting all of the output from your script and then running echo <that output>. Not exactly all that useful nor what you wanted.

The -n argument is how many items from the input to use with each command that gets run (nothing, by itself, about parallelism here).

To do what you want with xargs you would need to do something more like this (untested):

printf %s\\n {0..99} | xargs -n 1 -P 8 script-to-run.sh input/ output/

Which breaks down like this.

  • printf %s\\n {0..99} – Print one number per-line from 0 to 99.
  • Run xargs
    • taking at most one argument per run command line
    • and run up to eight processes at a time

More Related Contents:

  1. How do you run multiple programs in parallel from a bash script?
  2. Running multiple commands with xargs
  3. Calling shell functions with xargs
  4. How to ignore xargs commands if stdin input is empty?
  5. Running a limited number of child processes in parallel in bash? [duplicate]
  6. How to parallelize for-loop in bash limiting number of processes
  7. Is there a way to make bash job control quiet?
  8. Running several scripts in parallel bash script [duplicate]
  9. How to use > in an xargs command?
  10. Parallel wget in Bash [duplicate]
  11. Using grep to search for hex strings in a file
  12. Correct Bash and shell script variable capitalization
  13. Using getopts to process long and short command line options
  14. How do I compare two string variables in an ‘if’ statement in Bash? [duplicate]
  15. Brace expansion with a Bash variable – {0..$foo}
  16. “~/Desktop/test.txt: No such file or directory”
  17. Associative arrays: error “declare: -A: invalid option”
  18. How to put a line comment for a multi-line command [duplicate]
  19. How to calculate time elapsed in bash script?
  20. How do I get the directory where a Bash script is located from within the script itself?
  21. Bash: Syntax error: redirection unexpected
  22. Difference between return and exit in Bash functions
  23. How to kill all subprocesses of shell?
  24. Check if all of multiple strings or regexes exist in a file
  25. How to get the PID of a process that is piped to another process in Bash?
  26. How to reset COMP_WORDBREAKS without affecting other completion script?
  27. Extract version number from a string
  28. Rsync syntax error when run from bash script
  29. Store output of sed into a variable [duplicate]
  30. How to loop over files in natural order in Bash?

Leave a Comment