Django multiprocessing and database connections

by

Multiprocessing copies connection objects between processes because it forks processes, and therefore copies all the file descriptors of the parent process. That being said, a connection to the SQL server is just a file, you can see it in linux under /proc//fd/…. any open file will be shared between forked processes. You can find more about forking here.

My solution was just simply close db connection just before launching processes, each process recreate connection itself when it will need one (tested in django 1.4):

from django import db
db.connections.close_all()
def db_worker():      
    some_paralell_code()
Process(target = db_worker,args = ())

Pgbouncer/pgpool is not connected with threads in a meaning of multiprocessing. It’s rather solution for not closing connection on each request = speeding up connecting to postgres while under high load.

Update:

To completely remove problems with database connection simply move all logic connected with database to db_worker – I wanted to pass QueryDict as an argument… Better idea is simply pass list of ids… See QueryDict and values_list(‘id’, flat=True), and do not forget to turn it to list! list(QueryDict) before passing to db_worker. Thanks to that we do not copy models database connection.

def db_worker(models_ids):        
    obj = PartModelWorkerClass(model_ids) # here You do Model.objects.filter(id__in = model_ids)
    obj.run()


model_ids = Model.objects.all().values_list('id', flat=True)
model_ids = list(model_ids) # cast to list
process_count = 5
delta = (len(model_ids) / process_count) + 1

# do all the db stuff here ...

# here you can close db connection
from django import db
db.connections.close_all()

for it in range(0:process_count):
    Process(target = db_worker,args = (model_ids[it*delta:(it+1)*delta]))

More Related Contents:

  1. Django Cookies, how can I set them?
  2. Django: Populate user ID when saving a model
  3. Get model’s fields in Django
  4. Can django’s auth_user.username be varchar(75)? How could that be done?
  5. Manager isn’t available; User has been swapped for ‘pet.Person’
  6. How to get the list of the authenticated users?
  7. Caching query results in django
  8. Django delete FileField
  9. Raise a validation error in a model’s save method in Django
  10. Overriding admin css in django
  11. SocketException: OS Error: Connection refused, errno = 111 in flutter using django backend
  12. Django: show a ManyToManyField in a template?
  13. Django: ordering numerical value with order_by
  14. Access kwargs from a URL in a Django template
  15. How to register users in Django REST framework?
  16. How to do a multi-step form in Django?
  17. Example of Django Class-Based DeleteView
  18. Accelerate bulk insert using Django’s ORM?
  19. Django admin DoesNotExist at /admin/
  20. Pass request context to serializer from Viewset in Django Rest Framework
  21. Django: permission denied when trying to access database after restore (migration)
  22. UnicodeEncodeError: ‘ascii’ codec can’t encode character
  23. How can I upload and download files with graphene-django?
  24. Django give Error 500 for all static files like CSS and Images, when DEBUG is False
  25. Django’s forms.Form vs forms.ModelForm
  26. Django, query filtering from model method
  27. Parsing a Datetime String into a Django DateTimeField
  28. Forbidden (403) CSRF verification failed. Request aborted
  29. How can I test binary file uploading with django-rest-framework’s test client?
  30. Django – [Errno 111] Connection refused

Leave a Comment