I think this is most likely an aiohttp bug. Specifically, I found this issue on their github: https://github.com/aio-libs/aiohttp/issues/4324 I realise this doesn’t necessarily help you, but maybe you can switch back and stop banging your head against a wall. Your code is fine!
If I understand you well, you want to limit the number of simultaneous requests? There is a object inside asyncio named Semaphore, it works like an asynchronous RLock. semaphore = asyncio.Semaphore(50) #… async def limit_wrap(url): async with semaphore: # do what you want #… results = asyncio.gather([limit_wrap(url) for url in urls]) updated Suppose I make … Read more
Working example: import asyncio import aiohttp import ssl url_list = [‘https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530396000&before=1530436000’, ‘https://api.pushshift.io/reddit/search/comment/?q=Nestle&size=30&after=1530436000&before=1530476000’] async def fetch(session, url): async with session.get(url, ssl=ssl.SSLContext()) as response: return await response.json() async def fetch_all(urls, loop): async with aiohttp.ClientSession(loop=loop) as session: results = await asyncio.gather(*[fetch(session, url) for url in urls], return_exceptions=True) return results if __name__ == ‘__main__’: loop = asyncio.get_event_loop() urls = … Read more
Although it’s not exactly a limit on the number of requests per second, note that since v2.0, when using a ClientSession, aiohttp automatically limits the number of simultaneous connections to 100. You can modify the limit by creating your own TCPConnector and passing it into the ClientSession. For instance, to create a client limited to … Read more
One way would be to use create_task function: import asyncio async def handler_message(request): … loop = asyncio.get_event_loop() loop.create_task(perform_message(x,y,z)) … As per the loop documentation, starting Python 3.10, asyncio.get_event_loop() is deprecated. If you’re trying to get a loop instance from a coroutine/callback, you should use asyncio.get_running_loop() instead. This method will not work if called from the … Read more
Eventually I found an answer in this thread. The method I was looking for is run_in_executor. This allows a synchronous function to be run asynchronously without blocking an event loop. In the sleep example I posted above, it might look like this: import asyncio from time import sleep async def sleep_async(loop, delay): # None uses … Read more
You made faire_toutes_les_requetes_sans_bloquer an awaitable function, a coroutine, by using async def. When you call an awaitable function, you create a new coroutine object. The code inside the function won’t run until you then await on the function or run it as a task: >>> async def foo(): … print(“Running the foo coroutine”) … >>> … Read more
One way would be to use create_task function: import asyncio async def handler_message(request): … loop = asyncio.get_event_loop() loop.create_task(perform_message(x,y,z)) … As per the loop documentation, starting Python 3.10, asyncio.get_event_loop() is deprecated. If you’re trying to get a loop instance from a coroutine/callback, you should use asyncio.get_running_loop() instead. This method will not work if called from the … Read more
It’s pretty simple to delegate a method to a thread or sub-process using BaseEventLoop.run_in_executor: import asyncio import time from concurrent.futures import ProcessPoolExecutor def cpu_bound_operation(x): time.sleep(x) # This is some operation that is CPU-bound @asyncio.coroutine def main(): # Run cpu_bound_operation in the ProcessPoolExecutor # This will make your coroutine block, but won’t block # the event … Read more
To use requests (or any other blocking libraries) with asyncio, you can use BaseEventLoop.run_in_executor to run a function in another thread and yield from it to get the result. For example: import asyncio import requests @asyncio.coroutine def main(): loop = asyncio.get_event_loop() future1 = loop.run_in_executor(None, requests.get, ‘http://www.google.com’) future2 = loop.run_in_executor(None, requests.get, ‘http://www.google.co.uk’) response1 = yield from … Read more