It works using AsyncResult. (see this answer) So first create the task: from cel.tasks import add res = add.delay(3,4) print(res.status) # ‘SUCCESS’ print(res.id) # ‘432890aa-4f02-437d-aaca-1999b70efe8d’ Then start another python shell: from celery.result import AsyncResult from cel.tasks import app res = AsyncResult(‘432890aa-4f02-437d-aaca-1999b70efe8d’,app=app) print(res.state) # ‘SUCCESS’ print(res.get()) # 7
EDIT: 01/12/2021 previous answer (find it at the bottom) didn’t age well therefore I added a combination of possible solutions that may satisfy those who still look on how to co-use asyncio and Celery Lets quickly break up the use cases first (more in-depth analysis here: asyncio and coroutines vs task queues): If the task … Read more
First, let clarify the difference between celery library (which you get with pip install or in your setup.py) and celery worker – which is the actual process that dequeue tasks from the broker and handle them. Of course you might wanna have multiple workers/processes (for separating different task to a different worker – for example). … Read more
Here’s the code I’ve been using. celery.task.control.Inspect.stats() returns a dict containing lots of details about the currently available workers, None if there are no workers running, or raises an IOError if it can’t connect to the message broker. I’m using RabbitMQ – it’s possible that other messaging systems might behave slightly differently. This worked in … Read more
When your logger initialized in the beginning of “another module” it links to another logger. Which handle your messages. It can be root logger, or usually I see in Django projects – logger with name ”. Best way here, is overriding your logging config: LOGGING = { ‘version’: 1, ‘disable_existing_loggers’: True, ‘formatters’: { ‘simple’: { … Read more
A bit late here, but you should be able to define a custom encoder and decoder by registering them in the kombu serializer registry, as in the docs: http://docs.celeryproject.org/en/latest/userguide/calling.html#serializers. For example, the following is a custom datetime serializer/deserializer (subclassing python’s builtin json module) for Django: myjson.py (put it in the same folder of your settings.py … Read more
Okay here is how I got Scrapy working with my Django project that uses Celery for queuing up what to crawl. The actual workaround came primarily from joehillen’s code located here http://snippets.scrapy.org/snippets/13/ First the tasks.py file from celery import task @task() def crawl_domain(domain_pk): from crawl import domain_crawl return domain_crawl(domain_pk) Then the crawl.py file from multiprocessing … Read more
You can set task time limits (hard and/or soft) either while defining a task or while calling. from celery.exceptions import SoftTimeLimitExceeded @celery.task(time_limit=20) def mytask(): try: return do_work() except SoftTimeLimitExceeded: cleanup_in_a_hurry() or mytask.apply_async(args=[], kwargs={}, time_limit=30, soft_time_limit=10)
Prefetching can improve the performance. Workers don’t need to wait for the next message from a broker to process. Communicating with a broker once and processing a lot of messages gives a performance gain. Getting a message from a broker (even from a local one) is expensive compared to the local memory access. Workers are … Read more