Make sure only one worker launches the apscheduler event in a pyramid web app running multiple workers

by

Because Gunicorn is starting with 8 workers (in your example), this forks the app 8 times into 8 processes. These 8 processes are forked from the Master process, which monitors each of their status & has the ability to add/remove workers.

Each process gets a copy of your APScheduler object, which initially is an exact copy of your Master processes’ APScheduler. This results in each “nth” worker (process) executing each job a total of “n” times.

A hack around this is to run gunicorn with the following options:

env/bin/gunicorn module_containing_app:app -b 0.0.0.0:8080 --workers 3 --preload

The --preload flag tells Gunicorn to “load the app before forking the worker processes“. By doing so, each worker is “given a copy of the app, already instantiated by the Master, rather than instantiating the app itself”. This means the following code only executes once in the Master process:

rerun_monitor = Scheduler()
rerun_monitor.start()
rerun_monitor.add_interval_job(job_to_be_run,\
            seconds=JOB_INTERVAL)

Additionally, we need to set the jobstore to be anything other than :memory:.This way, although each worker is its own independent process unable of communicating with the other 7, by using a local database (rather then memory) we guarantee a single-point-of-truth for CRUD operations on the jobstore.

from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore

rerun_monitor = Scheduler(
    jobstores={'default': SQLAlchemyJobStore(url="sqlite:///jobs.sqlite")})
rerun_monitor.start()
rerun_monitor.add_interval_job(job_to_be_run,\
            seconds=JOB_INTERVAL)

Lastly, we want to use the BackgroundScheduler because of its implementation of start(). When we call start() in the BackgroundScheduler, a new thread is spun up in the background, which is responsible for scheduling/executing jobs. This is significant because remember in step (1), due to our --preload flag we only execute the start() function once, in the Master Gunicorn process. By definition, forked processes do not inherit the threads of their Parent, so each worker doesn’t run the BackgroundScheduler thread.

from apscheduler.jobstores.sqlalchemy import SQLAlchemyJobStore

rerun_monitor = BackgroundScheduler(
    jobstores={'default': SQLAlchemyJobStore(url="sqlite:///jobs.sqlite")})
rerun_monitor.start()
rerun_monitor.add_interval_job(job_to_be_run,\
            seconds=JOB_INTERVAL)

As a result of all of this, every Gunicorn worker has an APScheduler that has been tricked into a “STARTED” state, but actually isn’t running because it drops the threads of it’s parent! Each instance is also capable of updating the jobstore database, just not executing any jobs!

Check out flask-APScheduler for a quick way to run APScheduler in a web-server (like Gunicorn), and enable CRUD operations for each job.

More Related Contents:

  1. (13: Permission denied) while connecting to upstream:[nginx]
  2. How can I modify Procfile to run Gunicorn process in a non-standard folder on Heroku?
  3. ImportError: No module named django.core.wsgi Apache + VirtualEnv + AWS + WSGI
  4. Host Django on subfolder
  5. 110: Connection timed out (Nginx/Gunicorn)
  6. Extending the User model with custom fields in Django
  7. log all sql queries
  8. How to define two fields “unique” as couple
  9. Creating a extended user profile
  10. Django – iterate number in for loop of a template
  11. Track the number of “page views” or “hits” of an object?
  12. Django Broken pipe in Debug mode
  13. Django: Can class-based views accept two forms at a time?
  14. Django: Redirect logged in users from login page
  15. How do I add a Foreign Key Field to a ModelForm in Django?
  16. Django data migration when changing a field to ManyToMany
  17. CSRF Failed: CSRF token missing or incorrect
  18. Django templates: value of dictionary key with a space in it
  19. gunicorn autoreload on source change
  20. How can I list urlpatterns (endpoints) on Django?
  21. how to show datepicker calendar on datefield
  22. Order by count of a ForeignKey field?
  23. Django removing object from ManyToMany relationship
  24. Log in / Sign up directly on home page
  25. What is the right way to validate if an object exists in a django view without returning 404?
  26. Django vs other Python web frameworks?
  27. Django Order By Date, but have “None” at end?
  28. How to limit file types on file uploads for ModelForms with FileFields?
  29. Django: access the parent instance from the Inline model admin
  30. django : Serving static files through nginx

Leave a Comment