The & has a special meaning in the shell. Escape it with \ or put the url in quotes to avoid this problem. wget http://search.yahoo.com/404handler?src=search\&p=food+delicious -O test.html or wget “http://search.yahoo.com/404handler?src=search&p=food+delicious” -O test.html In many Unix shells, putting an & after a command causes it to be executed in the background.
There are several approaches to rate limiting; one of them is token bucket, for which you can find a recipe here and another one here. Usually you would want to do throttling or rate limiting on socket.send() and socket.recv(). You could play with socket-throttle and see if it does what you need. This is not … Read more
Much preferrable to pushing wget into the background using & or -b, you can use xargs to the same effect, and better. The advantage is that xargs will synchronize properly with no extra work. Which means that you are safe to access the downloaded files (assuming no error occurs). All downloads will have completed (or … Read more
For all users of the system via the /etc/wgetrc or for the user only with the ~/.wgetrc file: use_proxy=yes http_proxy=127.0.0.1:8080 https_proxy=127.0.0.1:8080 or via -e options placed after the URL: wget … -e use_proxy=yes -e http_proxy=127.0.0.1:8080 …
You can’t use wildcards in wget but the -A flag should work. From the wget manpage: You want to download all the gifs from a directory on an http server. You tried wget http://www.server.com/dir/*.gif, but that didn’t work because http retrieval does not support globbing. In that case, use: wget -r -l1 –no-parent -A.gif http://www.server.com/dir/ … Read more
To filter for specific file extensions: wget -A pdf,jpg -m -p -E -k -K -np http://site/path/ Or, if you prefer long option names: wget –accept pdf,jpg –mirror –page-requisites –adjust-extension –convert-links –backup-converted –no-parent http://site/path/ This will mirror the site, but the files without jpg or pdf extension will be automatically removed.
I’m not sure what wget is, but to get a file from the web and store it locally, you can use NSData: NSString *stringURL = @”http://www.somewhere.com/thefile.png”; NSURL *url = [NSURL URLWithString:stringURL]; NSData *urlData = [NSData dataWithContentsOfURL:url]; if ( urlData ) { NSArray *paths = NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES); NSString *documentsDirectory = [paths objectAtIndex:0]; NSString *filePath = … Read more
wget is capable of doing what you are asking. Just try the following: wget -p -k http://www.example.com/ The -p will get you all the required elements to view the site correctly (css, images, etc). The -k will change all links (to include those for CSS & images) to allow you to view the page offline … Read more
You could use a thread pool to download files in parallel: #!/usr/bin/env python3 from multiprocessing.dummy import Pool # use threads for I/O bound tasks from urllib.request import urlretrieve urls = […] result = Pool(4).map(urlretrieve, urls) # download 4 files at a time You could also download several files at once in a single thread using … Read more
You could tell wget to not download the contents in a couple of different ways: wget –spider http://www.example.com/cronit.php which will just perform a HEAD request but probably do what you want wget -O /dev/null http://www.example.com/cronit.php which will save the output to /dev/null (a black hole) You might want to look at wget’s -q switch too … Read more