Python requests, how to limit received size, transfer rate, and/or total time?

by

You could try setting stream=True, then aborting a request when your time or size limits are exceeded while you read the data in chunks.

As of requests release 2.3.0 the timeout applies to streaming requests too, so all you need to do is allow for a timeout for the initial connection and each iteration step:

r = requests.get(..., stream=True, timeout=initial_timeout)
r.raise_for_status()

if int(r.headers.get('Content-Length')) > your_maximum:
    raise ValueError('response too large')

size = 0
start = time.time()

for chunk in r.iter_content(1024):
    if time.time() - start > receive_timeout:
        raise ValueError('timeout reached')

    size += len(chunk)
    if size > your_maximum:
        raise ValueError('response too large')

    # do something with chunk

Adjust the timeout as needed.

For requests releases < 2.3.0 (which included this change) you could not time out the r.iter_content() yield; a server that stops responding in the middle of a chunk would still tie up the connection. You’d have to wrap the above code in an additional timeout function to cut off long-running responses early.

More Related Contents:

  1. How to POST JSON data with Python Requests?
  2. How to install packages offline?
  3. How to send a “multipart/form-data” with requests in python?
  4. Sending “User-agent” using Requests library in Python
  5. Using python Requests with javascript pages
  6. How do I disable log messages from the Requests library?
  7. How to get Python requests to trust a self signed SSL certificate?
  8. Timeout for python requests.get entire response
  9. Python: download a file from an FTP server
  10. Python requests.exceptions.SSLError: EOF occurred in violation of protocol
  11. Python requests SSL error – certificate verify failed
  12. Download and save PDF file with Python requests module
  13. Python ‘requests’ library – define specific DNS?
  14. requests: how to disable / bypass proxy
  15. How can I send an xml body using requests library?
  16. Putting a `Cookie` in a `CookieJar`
  17. Requests library: missing SSL handshake certificates file after cx_Freeze
  18. Why doesn’t requests.get() return? What is the default timeout that requests.get() uses?
  19. google search with python requests library
  20. SSL error with Python requests despite up-to-date dependencies
  21. Python Requests with wincertstore
  22. Python requests speed up using keep-alive
  23. python requests.get always get 404
  24. What is the proper way to make downstream Https requests inside of Uvicorn/FastAPI?
  25. how to post multiple value with same key in python requests?
  26. How to limit download rate of HTTP requests in requests python library?
  27. Progress of Python requests post
  28. Requests with multiple connections
  29. Unable to get local issuer certificate when using requests
  30. Sending SOAP request using Python Requests

Leave a Comment