Can you perform multi-threaded tasks within Django?

by

As shown in this answer you can use the threading package to perform an asynchronous task. Everyone seems to recommend Celery, but it is often overkill for performing simple but long running tasks. I think it’s actually easier and more transparent to use threading.

Here’s a simple example for asyncing a crawler:

#views.py
import threading
from .models import Crawl

def startCrawl(request):
    task = Crawl()
    task.save()
    t = threading.Thread(target=doCrawl,args=[task.id])
    t.setDaemon(True)
    t.start()
    return JsonResponse({'id':task.id})

def checkCrawl(request,id):
    task = Crawl.objects.get(pk=id)
    return JsonResponse({'is_done':task.is_done, result:task.result})

def doCrawl(id):
    task = Crawl.objects.get(pk=id)
    # Do crawling, etc.

    task.result = result
    task.is_done = True
    task.save()

Your front end can make a request for startCrawl to start the crawl, it can make an Ajax request to check on it with checkCrawl which will return true and the result when it’s finished.

Update for Python3:

The documentation for the threading library recommends passing the daemon property as a keyword argument rather than using the setter:

t = threading.Thread(target=doCrawl,args=[task.id],daemon=True)
t.start()

Update for Python <3.7:

As discussed here, this bug can cause a slow memory leak that can overflow a long running server. The bug was fixed for Python 3.7 and above.

More Related Contents:

  1. error: can’t start new thread
  2. ‘pip’ is not recognized as an internal or external command
  3. What is the difference between null=True and blank=True in Django?
  4. Make 2 functions run at the same time
  5. Using Django time/date widgets in custom form
  6. Thread Safety in Python’s dictionary
  7. Django REST Framework upload image: “The submitted data was not a file”
  8. Difference between static STATIC_URL and STATIC_ROOT on Django
  9. Django Query That Get Most Recent Objects From Different Categories
  10. Django rest framework serializing many to many field
  11. How to compare two JSON objects with the same elements in a different order equal?
  12. How to replace Django’s primary key with a different integer that is unique for that table
  13. Best practices for adding .gitignore file for Python projects? [closed]
  14. How do I get the object if it exists, or None if it does not exist in Django?
  15. Find object in list that has attribute equal to some value (that meets any condition)
  16. How to encode UTF8 filename for HTTP headers? (Python, Django)
  17. django urls without a trailing slash do not redirect
  18. Error: “dictionary update sequence element #0 has length 1; 2 is required” on Django 1.4
  19. Configure Django and Google Cloud Storage?
  20. AttributeError: ‘module’ object has no attribute ‘setdefaultencoding’
  21. How to filter objects for count annotation in Django?
  22. Distributed task queues (Ex. Celery) vs crontab scripts
  23. How to redirect with messages to display them in Django Templates?
  24. WARNING: Running pip as the ‘root’ user
  25. Tkinter GUI, I/O & Threading: When to use queues, when events?
  26. Using a global dictionary with threads in Python
  27. InterfaceError (0, ”)
  28. Python thread name doesn’t show up on ps or htop
  29. Django Admin Page missing CSS
  30. Django 1.9 ImportError for import_module

Leave a Comment