celeryd

Setting Time Limit on specific task with celery

You can set task time limits (hard and/or soft) either while defining a task or while calling. from celery.exceptions import SoftTimeLimitExceeded @celery.task(time_limit=20) def mytask(): try: return do_work() except SoftTimeLimitExceeded: cleanup_in_a_hurry() or mytask.apply_async(args=[], kwargs={}, time_limit=30, soft_time_limit=10)

Understanding celery task prefetching

Prefetching can improve the performance. Workers don’t need to wait for the next message from a broker to process. Communicating with a broker once and processing a lot of messages gives a performance gain. Getting a message from a broker (even from a local one) is expensive compared to the local memory access. Workers are … Read more