Python: HTTP Post a large file with streaming

by

Reading through the mailing list thread linked to by systempuntoout, I found a clue towards the solution.

The mmap module allows you to open file that acts like a string. Parts of the file are loaded into memory on demand.

Here’s the code I’m using now:

import urllib2
import mmap

# Open the file as a memory mapped string. Looks like a string, but 
# actually accesses the file behind the scenes. 
f = open('somelargefile.zip','rb')
mmapped_file_as_string = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)

# Do the request
request = urllib2.Request(url, mmapped_file_as_string)
request.add_header("Content-Type", "application/zip")
response = urllib2.urlopen(request)

#close everything
mmapped_file_as_string.close()
f.close()

More Related Contents:

  1. Using an HTTP PROXY – Python [duplicate]
  2. Python-Requests close http connection
  3. Python urllib2 with keep alive
  4. Python urllib2 Progress Hook
  5. Python urllib2.urlopen() is slow, need a better way to read several urls
  6. What should I do if socket.setdefaulttimeout() is not working?
  7. Unescape Python Strings From HTTP
  8. How do I catch a specific HTTP error in Python?
  9. What is the fastest way to send 100,000 HTTP requests in Python?
  10. Import error: No module name urllib2
  11. Python Requests library redirect new url
  12. python ignore certificate validation urllib2
  13. Simple URL GET/POST function in Python
  14. Only download a part of the document using python requests
  15. How to get response SSL certificate from requests in python?
  16. Read timeout using either urllib2 or any other http library
  17. open url from pythonanywhere
  18. How to send zip files in the python Flask framework?
  19. Submitting to a web form using python
  20. urllib2 HTTP Error 400: Bad Request
  21. Limiting number of processes in multiprocessing python
  22. Checking if a website is up via Python
  23. How to get the raw content of a response in requests with Python?
  24. Ideal method for sending multiple HTTP requests over Python? [duplicate]
  25. Why do I get “‘str’ object has no attribute ‘read'” when trying to use `json.load` on a string? [duplicate]
  26. how to follow meta refreshes in Python
  27. Python seek on remote file using HTTP
  28. Multiprocessing useless with urllib2?
  29. How to resolve URLError:
  30. Python requests.exception.ConnectionError: connection aborted “BadStatusLine”

Leave a Comment