Stream large binary files with urllib2 to file

by

No reason to work line by line (small chunks AND requires Python to find the line ends for you!-), just chunk it up in bigger chunks, e.g.:

# from urllib2 import urlopen # Python 2
from urllib.request import urlopen # Python 3

response = urlopen(url)
CHUNK = 16 * 1024
with open(file, 'wb') as f:
    while True:
        chunk = response.read(CHUNK)
        if not chunk:
            break
        f.write(chunk)

Experiment a bit with various CHUNK sizes to find the “sweet spot” for your requirements.

More Related Contents:

  1. Using MultipartPostHandler to POST form-data with Python
  2. Download and decompress gzipped file in memory?
  3. how to can i go over a file twice?
  4. open() gives FileNotFoundError/IOError: Errno 2 No such file or directory
  5. How do I get the path and name of the file that is currently executing?
  6. How can I use a SOCKS 4/5 proxy with urllib2?
  7. How to stream an HttpResponse with Django
  8. How can I separate the functions of a class into multiple files?
  9. How to find out whether a file is at its `eof`?
  10. Python: HTTP Post a large file with streaming
  11. How come a file doesn’t get written until I stop the program?
  12. python: urllib2 how to send cookie with urlopen request
  13. Reading two text files line by line simultaneously
  14. filename and line number of Python script
  15. Search for a file using a wildcard
  16. Python append multiple files in given order to one big file
  17. What should I do if socket.setdefaulttimeout() is not working?
  18. Save a large file using the Python requests library [duplicate]
  19. How to add a key-value to JSON data retrieved from a file?
  20. How does python close files that have been gc’ed?
  21. Why do I get “Pickle – EOFError: Ran out of input” reading an empty file?
  22. How do I wrap a string in a file in Python?
  23. Making HTTP HEAD request with urllib2 from Python 2
  24. “TypeError: a bytes-like object is required, not ‘str'” when handling file content in Python 3
  25. UnicodeDecodeError in Python when reading a file, how to ignore the error and jump to the next line?
  26. For loop not working twice on the same file descriptor [duplicate]
  27. f.read coming up empty
  28. Why do I get “‘str’ object has no attribute ‘read'” when trying to use `json.load` on a string? [duplicate]
  29. Multiprocessing useless with urllib2?
  30. Best way to choose a random file from a directory

Leave a Comment