When to use and when not to use Python 3.5 `await` ?

by

By default all your code is synchronous. You can make it asynchronous defining functions with async def and “calling” these functions with await. A More correct question would be “When should I write asynchronous code instead of synchronous?”. Answer is “When you can benefit from it”. In cases when you work with I/O operations as you noted you will usually benefit:

# Synchronous way:
download(url1)  # takes 5 sec.
download(url2)  # takes 5 sec.
# Total time: 10 sec.

# Asynchronous way:
await asyncio.gather(
    async_download(url1),  # takes 5 sec. 
    async_download(url2)   # takes 5 sec.
)
# Total time: only 5 sec. (+ little overhead for using asyncio)

Of course, if you created a function that uses asynchronous code, this function should be asynchronous too (should be defined as async def). But any asynchronous function can freely use synchronous code. It makes no sense to cast synchronous code to asynchronous without some reason:

# extract_links(url) should be async because it uses async func async_download() inside
async def extract_links(url):  

    # async_download() was created async to get benefit of I/O
    html = await async_download(url)  

    # parse() doesn't work with I/O, there's no sense to make it async
    links = parse(html)  

    return links

One very important thing is that any long synchronous operation (> 50 ms, for example, it’s hard to say exactly) will freeze all your asynchronous operations for that time:

async def extract_links(url):
    data = await download(url)
    links = parse(data)
    # if search_in_very_big_file() takes much time to process,
    # all your running async funcs (somewhere else in code) will be frozen
    # you need to avoid this situation
    links_found = search_in_very_big_file(links)

You can avoid it calling long running synchronous functions in separate process (and awaiting for result):

executor = ProcessPoolExecutor(2)

async def extract_links(url):
    data = await download(url)
    links = parse(data)
    # Now your main process can handle another async functions while separate process running    
    links_found = await loop.run_in_executor(executor, search_in_very_big_file, links)

One more example: when you need to use requests in asyncio. requests.get is just synchronous long running function, which you shouldn’t call inside async code (again, to avoid freezing). But it’s running long because of I/O, not because of long calculations. In that case, you can use ThreadPoolExecutor instead of ProcessPoolExecutor to avoid some multiprocessing overhead:

executor = ThreadPoolExecutor(2)

async def download(url):
    response = await loop.run_in_executor(executor, requests.get, url)
    return response.text

More Related Contents:

  1. “Fire and forget” python async/await
  2. How can I periodically execute a function with asyncio?
  3. How to use asyncio with existing blocking library?
  4. “Asyncio Event Loop is Closed” when getting loop
  5. How to use ‘yield’ inside async function?
  6. Does asyncio supports asynchronous I/O for file operations?
  7. asyncio.ensure_future vs. BaseEventLoop.create_task vs. simple coroutine?
  8. Asynchronous method call in Python?
  9. Python 3.7 – asyncio.sleep() and time.sleep()
  10. RuntimeError: This event loop is already running in python
  11. Python type hinting without cyclic imports
  12. When using asyncio, how do you allow all running tasks to finish before shutting down the event loop
  13. Sharing python objects across multiple workers
  14. Timeout for python coroutines
  15. Learning asyncio: “coroutine was never awaited” warning error
  16. Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)
  17. What kind of problems (if any) would there be combining asyncio with multiprocessing?
  18. How can I wrap a synchronous function in an async coroutine?
  19. What does “SSLError: [SSL] PEM lib (_ssl.c:2532)” mean using the Python ssl library?
  20. asyncio.sleep() vs time.sleep() in Python
  21. Mocking async call in python 3.5
  22. multiprocessing vs multithreading vs asyncio
  23. Why am I getting “invalid syntax” from an f-string? [duplicate]
  24. Use asyncio and Tkinter (or another GUI lib) together without freezing the GUI
  25. typing.Any vs object?
  26. Make a Python asyncio call from a Flask route
  27. Create generator that yields coroutine results as the coroutines finish
  28. Is it possible to run only a single step of the asyncio event loop
  29. Handling Timeouts with asyncio
  30. How to combine Celery with asyncio?

Leave a Comment