aiohttp: rate limiting parallel requests

by

If I understand you well, you want to limit the number of simultaneous requests?

There is a object inside asyncio named Semaphore, it works like an asynchronous RLock.

semaphore = asyncio.Semaphore(50)
#...
async def limit_wrap(url):
    async with semaphore:
        # do what you want
#...
results = asyncio.gather([limit_wrap(url) for url in urls])

updated

Suppose I make 50 concurrent requests, and they all finish in 2 seconds. So, it doesn’t touch the limitation(only 25 requests per seconds).

That means I should make 100 concurrent requests, and they all finish in 2 seconds too(50 requests per seconds). But before you actually make those requests, how could you determine how long will they finish?

Or if you doesn’t mind finished requests per second but requests made per second. You can:

async def loop_wrap(urls):
    for url in urls:
        asyncio.ensure_future(download(url))
        await asyncio.sleep(1/50)

asyncio.ensure_future(loop_wrap(urls))
loop.run_forever()

The code above will create a Future instance every 1/50 second.

More Related Contents:

  1. How to combine python asyncio with threads?
  2. How can I call an async function without await?
  3. Learning asyncio: “coroutine was never awaited” warning error
  4. How can I wrap a synchronous function in an async coroutine?
  5. How to call an async function without await?
  6. aiohttp: set maximum number of requests per second
  7. Exception event loop is closed with aiohttp and asyncio in python 3.8
  8. When to use and when not to use Python 3.5 `await` ?
  9. What are the differences between the threading and multiprocessing modules?
  10. How could I use requests in asyncio?
  11. How to limit concurrency with Python asyncio?
  12. Meaning of inter_op_parallelism_threads and intra_op_parallelism_threads
  13. Parallelize apply after pandas groupby
  14. Poor scaling of multiprocessing Pool.map() on a list of large objects: How to achieve better parallel scaling in python?
  15. Parallel Processing in python
  16. Can an asyncio event loop run in the background without suspending the Python interpreter?
  17. How do I run Python asyncio code in a Jupyter notebook?
  18. How to test Python 3.4 asyncio code?
  19. How to use ‘yield’ inside async function?
  20. Modify object in python multiprocessing
  21. Processes stuck in loop with PyInstaller-executable
  22. asyncio: Is it possible to cancel a future been run by an Executor?
  23. Why can I only use the await keyword inside of async function?
  24. How to do multiprocessing in FastAPI
  25. multiprocessing pool example does not work and freeze the kernel
  26. How to set class attribute with await in __init__
  27. How can I await inside future-like object’s __await__?
  28. cython shared memory in cython.parallel.prange – block
  29. Prompt for user input using python asyncio.create_server instance
  30. Proper way to use multiprocessor.Pool in a nested loop

Leave a Comment