Python Requests vs PyCurl Performance

by

I wrote you a full benchmark, using a trivial Flask application backed by gUnicorn/meinheld + nginx (for performance and HTTPS), and seeing how long it takes to complete 10,000 requests. Tests are run in AWS on a pair of unloaded c4.large instances, and the server instance was not CPU-limited.

TL;DR summary: if you’re doing a lot of networking, use PyCurl, otherwise use requests. PyCurl finishes small requests 2x-3x as fast as requests until you hit the bandwidth limit with large requests (around 520 MBit or 65 MB/s here), and uses from 3x to 10x less CPU power. These figures compare cases where connection pooling behavior is the same; by default, PyCurl uses connection pooling and DNS caches, where requests does not, so a naive implementation will be 10x as slow.

Combined-chart-RPS
CPU Time by request size detailed

Just HTTP throughput
Just HTTP RPS

Note that double log plots are used for the below graph only, due to the orders of magnitude involved
HTTP & HTTPS throughput
HTTP & HTTPS RPS

  • pycurl takes about 73 CPU-microseconds to issue a request when reusing a connection
  • requests takes about 526 CPU-microseconds to issue a request when reusing a connection
  • pycurl takes about 165 CPU-microseconds to open a new connection and issue a request (no connection reuse), or ~92 microseconds to open
  • requests takes about 1078 CPU-microseconds to open a new connection and issue a request (no connection reuse), or ~552 microseconds to open

Full results are in the link, along with the benchmark methodology and system configuration.

Caveats: although I’ve taken pains to ensure the results are collected in a scientific way, it’s only testing one system type and one operating system, and a limited subset of performance and especially HTTPS options.

More Related Contents:

  1. Why does Python code run faster in a function?
  2. Why is looping over range() in Python faster than using a while loop?
  3. How do I get the IP address from a http request using the requests library?
  4. python: deque vs list performance comparison
  5. Python Slice Assignment Memory Usage
  6. Python Numpy Data Types Performance
  7. Is doing multiplication multiple times or assigning to a variable faster?
  8. How to send a “multipart/form-data” with requests in python?
  9. Sending “User-agent” using Requests library in Python
  10. pandas loc vs. iloc vs. at vs. iat?
  11. What is the best way to generate all possible three letter strings?
  12. How do I disable log messages from the Requests library?
  13. Why is numpy’s einsum faster than numpy’s built in functions?
  14. Speed comparison with Project Euler: C vs Python vs Erlang vs Haskell
  15. Cost of exception handlers in Python
  16. File read using “open()” vs “with open()” [duplicate]
  17. Download and save PDF file with Python requests module
  18. [] and {} vs list() and dict(), which is better?
  19. Making an API call in Python with an API that requires a bearer token
  20. requests: how to disable / bypass proxy
  21. Putting a `Cookie` in a `CookieJar`
  22. Requests library: missing SSL handshake certificates file after cx_Freeze
  23. multiprocessing.Pool() slower than just using ordinary functions
  24. Complexity of list.index(x) in Python
  25. Python requests speed up using keep-alive
  26. Change the connection pool size for Python’s “requests” module when in Threading
  27. Python requests, how to limit received size, transfer rate, and/or total time?
  28. differences between “d = dict()” and “d = {}”
  29. Python 3: using requests does not get the full content of a web page
  30. numpy faster than numba and cython , how to improve numba code

Leave a Comment