Parallel wget in Bash [duplicate]

by

Much preferrable to pushing wget into the background using & or -b, you can use xargs to the same effect, and better.

The advantage is that xargs will synchronize properly with no extra work. Which means that you are safe to access the downloaded files (assuming no error occurs). All downloads will have completed (or failed) once xargs exits, and you know by the exit code whether all went well. This is much preferrable to busy waiting with sleep and testing for completion manually.

Assuming that URL_LIST is a variable containing all the URLs (can be constructed with a loop in the OP’s example, but could also be a manually generated list), running this:

echo $URL_LIST | xargs -n 1 -P 8 wget -q

will pass one argument at a time (-n 1) to wget, and execute at most 8 parallel wget processes at a time (-P 8). xarg returns after the last spawned process has finished, which is just what we wanted to know. No extra trickery needed.

The “magic number” of 8 parallel downloads that I’ve chosen is not set in stone, but it is probably a good compromise. There are two factors in “maximising” a series of downloads:

One is filling “the cable”, i.e. utilizing the available bandwidth. Assuming “normal” conditions (server has more bandwidth than client), this is already the case with one or at most two downloads. Throwing more connections at the problem will only result in packets being dropped and TCP congestion control kicking in, and N downloads with asymptotically 1/N bandwidth each, to the same net effect (minus the dropped packets, minus window size recovery). Packets being dropped is a normal thing to happen in an IP network, this is how congestion control is supposed to work (even with a single connection), and normally the impact is practically zero. However, having an unreasonably large number of connections amplifies this effect, so it can be come noticeable. In any case, it doesn’t make anything faster.

The second factor is connection establishment and request processing. Here, having a few extra connections in flight really helps. The problem one faces is the latency of two round-trips (typically 20-40ms within the same geographic area, 200-300ms inter-continental) plus the odd 1-2 milliseconds that the server actually needs to process the request and push a reply to the socket. This is not a lot of time per se, but multiplied by a few hundred/thousand requests, it quickly adds up.
Having anything from half a dozen to a dozen requests in-flight hides most or all of this latency (it is still there, but since it overlaps, it does not sum up!). At the same time, having only a few concurrent connections does not have adverse effects, such as causing excessive congestion, or forcing a server into forking new processes.

More Related Contents:

  1. How do you run multiple programs in parallel from a bash script?
  2. Running a limited number of child processes in parallel in bash? [duplicate]
  3. How to parallelize for-loop in bash limiting number of processes
  4. Is there a way to make bash job control quiet?
  5. Running programs in parallel using xargs
  6. Running several scripts in parallel bash script [duplicate]
  7. How can I check if a program exists from a Bash script?
  8. How can I compare numbers in Bash?
  9. echo “#!” fails — “event not found”
  10. Pipe output and capture exit status in Bash
  11. Redirect stderr and stdout in Bash [duplicate]
  12. Which characters need to be escaped when using Bash?
  13. bash : Illegal number
  14. Shell redirection i/o order
  15. Loop through all the files with a specific extension
  16. Colorized grep — viewing the entire file with highlighted matches
  17. How to pass in password to pg_dump?
  18. How can I profile a Bash shell script slow startup?
  19. Select random lines from a file
  20. Variables getting reset after the while read loop that reads from a pipeline
  21. How to delete and replace last line in the terminal using bash?
  22. Unset readonly variable in bash
  23. Should I use a Shebang with Bash scripts?
  24. How to use arguments from previous command?
  25. How to remove last n characters from a string in Bash?
  26. access host’s ssh tunnel from docker container
  27. Command to escape a string in bash
  28. Getting “sed error – illegal byte sequence” (in bash) [duplicate]
  29. Indenting heredocs with spaces
  30. Subtract two variables in Bash

Leave a Comment