Running “unique” tasks with celery

by

Based on MattH’s answer, you could use a decorator like this:

def single_instance_task(timeout):
    def task_exc(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            lock_id = "celery-single-instance-" + func.__name__
            acquire_lock = lambda: cache.add(lock_id, "true", timeout)
            release_lock = lambda: cache.delete(lock_id)
            if acquire_lock():
                try:
                    func(*args, **kwargs)
                finally:
                    release_lock()
        return wrapper
    return task_exc

then, use it like so…

@periodic_task(run_every=timedelta(minutes=1))
@single_instance_task(60*10)
def fetch_articles()
    yada yada...

More Related Contents:

  1. Cancel an already executing task with Celery?
  2. Reporting yielded results of long-running Celery task
  3. How can I schedule a Task to execute at a specific time using celery?
  4. Distributed task queues (Ex. Celery) vs crontab scripts
  5. Running Scrapy spiders in a Celery task
  6. Django Celery Logging Best Practice
  7. Detect whether Celery is Available/Running
  8. What is the difference between null=True and blank=True in Django?
  9. Iterate over model instance field names and values in template
  10. Using Django time/date widgets in custom form
  11. Django rest framework serializing many to many field
  12. How to compare two JSON objects with the same elements in a different order equal?
  13. Best practices for adding .gitignore file for Python projects? [closed]
  14. How do I get the object if it exists, or None if it does not exist in Django?
  15. Find object in list that has attribute equal to some value (that meets any condition)
  16. Error: “dictionary update sequence element #0 has length 1; 2 is required” on Django 1.4
  17. Configure Django and Google Cloud Storage?
  18. AttributeError: ‘module’ object has no attribute ‘setdefaultencoding’
  19. CSS not loading wrong MIME type Django [duplicate]
  20. How to filter objects for count annotation in Django?
  21. Django populate() isn’t reentrant
  22. Change Django ModelChoiceField to show users’ full names rather than usernames
  23. django-debug-toolbar not showing up
  24. gunicorn.errors.HaltServer: django
  25. Django Admin Page missing CSS
  26. How do I package a python application to make it pip-installable?
  27. How do I start up remote debugging with PyCharm?
  28. is not JSON serializable
  29. Django: sudden DB reset after Heroku sleep
  30. How to run Debug server for Django project in PyCharm Community Edition?

Leave a Comment