python requests module and connection reuse

by

Global functions like requests.get or requests.post create the requests.Session instance on each call. Connections made with these functions cannot be reused, because you cannot access automatically created session and use it’s connection pool for subsequent requests. It’s fine to use these functions if you have to do just a few requests. Otherwise you’ll want to manage sessions yourself.

Here is a quick display of requests behavior when you use global get function and session.

Preparation, not really relevant to the question:

>>> import logging, requests, timeit
>>> logging.basicConfig(level=logging.DEBUG, format="%(message)s")

See, a new connection is established each time you call get:

>>> _ = requests.get("https://www.wikipedia.org")
Starting new HTTPS connection (1): www.wikipedia.org
>>> _ = requests.get("https://www.wikipedia.org")
Starting new HTTPS connection (1): www.wikipedia.org

But if you use the same session for subsequent calls, the connection gets reused:

>>> session = requests.Session()
>>> _ = session.get("https://www.wikipedia.org")
Starting new HTTPS connection (1): www.wikipedia.org
>>> _ = session.get("https://www.wikipedia.org")
>>> _ = session.get("https://www.wikipedia.org")
>>> _ = session.get("https://www.wikipedia.org")

Performance:

>>> timeit.timeit('_ = requests.get("https://www.wikipedia.org")', 'import requests', number=100)
Starting new HTTPS connection (1): www.wikipedia.org
Starting new HTTPS connection (1): www.wikipedia.org
Starting new HTTPS connection (1): www.wikipedia.org
...
Starting new HTTPS connection (1): www.wikipedia.org
Starting new HTTPS connection (1): www.wikipedia.org
Starting new HTTPS connection (1): www.wikipedia.org
52.74904417991638
>>> timeit.timeit('_ = session.get("https://www.wikipedia.org")', 'import requests; session = requests.Session()', number=100)
Starting new HTTPS connection (1): www.wikipedia.org
15.770191192626953

Works much faster when you reuse the session (and thus session’s connection pool).

More Related Contents:

  1. Download large file in python with requests
  2. How to download image using requests
  3. How to upload file with python requests?
  4. What are the differences between the urllib, urllib2, urllib3 and requests module?
  5. How can I see the entire HTTP request that’s being sent by my Python application?
  6. How to use Python requests to fake a browser visit a.k.a and generate User Agent?
  7. ImportError: No module named requests
  8. Using headers with the Python requests library’s get method
  9. Requests (Caused by SSLError(“Can’t connect to HTTPS URL because the SSL module is not available.”) Error in PyCharm requesting website
  10. Web scraping program cannot find element which I can see in the browser
  11. Python urllib2 with keep alive
  12. How to load all entries in an infinite scroll at once to parse the HTML in python
  13. How do I get the IP address from a http request using the requests library?
  14. Adding headers to requests module
  15. Python Requests package: Handling xml response
  16. Python Requests vs PyCurl Performance
  17. Can Python Requests library be used on Google App Engine?
  18. python how to decode http response
  19. How to implement retry mechanism into python requests library?
  20. Using cookies.txt file with Python Requests
  21. Python requests with multithreading
  22. Python: FastAPI error 422 with POST request when sending JSON data
  23. How to change tcp keepalive timer using python script?
  24. Use python requests to download CSV
  25. How to get around python requests SSL and proxy error?
  26. requests.get returns 403 while the same url works in browser
  27. Http Redirection code 3XX in python requests
  28. Unit testing a python app that uses the requests library
  29. Difference between using requests.get() and requests.session().get()?
  30. Python 3: using requests does not get the full content of a web page

Leave a Comment