Background Worker with Flask

by

The downside of your approach is that there are many ways it could fail especially around stopping and restarting your flask application.

  • You will have to deal with graceful shutdown to give your worker a chance to finish its current task.
  • Sometime your worker won’t stop on time and might linger while you start another one when you reboot your flask application.

Here are some approches I would suggest depending on your constraints:

script + crontab

You only have to write a script that does whatever task you want and cron will take care of running it for you every few minutes. Advantages: cron will run it for you periodically and will start when the system starts. Disadvantages: if the task takes too long, you might have multiple instances of your script running at the same time. You can find some solutions for this problem here.

supervisord

supervisord is a neat way to deal with different daemons. You can set it to run your app, your background script or both and have them start with the server. Only downside is that you have to install supervisord and make sure its daemon is running when the server starts.

uwsgi

uwsgi is a very common way for deploying flask applications. It has few features that might come in handy for managing background workers.

Celery

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well. I think this is the best solution for scheduling background tasks for a flask application or any other python based application. But using it comes with some extra bulk. You will be introducing at least the following processes:
– a broker (rabbitmq or redis)
– a worker
– a scheduler

You can also get supervisord to manage all of the processes above and get them to start when the server starts.

Conclusion

In your quest of reducing the number of processes, I would highly suggest the crontab based solution as it can get you a long way. But please make sure your background script leaves an execution trace or logs of some sort.

More Related Contents:

  1. How to serve static files in Flask
  2. Flask view return error “View function did not return a response”
  3. Testing code that requires a Flask app or request context
  4. How do I get the different parts of a Flask request’s url?
  5. Why use Flask’s url_for?
  6. Remove file after Flask serves it
  7. Making an asynchronous task in Flask
  8. Send data from a textbox into Flask?
  9. Serve image stored in SQLAlchemy LargeBinary column
  10. Flask view shows 400 error instead of template with form
  11. Create dynamic arguments for url_for in Flask
  12. How to iterate through a list of dictionaries in Jinja template?
  13. Flask jsonify a list of objects
  14. Python flask app routing in cpanel: can only access root url
  15. demystify Flask app.secret_key
  16. ImportError: No module named MySQLdb
  17. Flask throwing ‘working outside of request context’ when starting sub thread
  18. How to run Flask app with Tornado
  19. Run Flask CLI command with PyCharm debugger
  20. Get list of all routes defined in the Flask app
  21. Flask App: Update progress bar while function runs
  22. Get inserted key before commit session
  23. Where do I define the domain to be used by url_for() in Flask?
  24. How to run Flask with Gunicorn in multithreaded mode
  25. Flask isn’t getting the checkbox value
  26. Make a Python asyncio call from a Flask route
  27. How to generate dynamic urls in flask?
  28. How to install Flask on Windows?
  29. Static files in Flask – robot.txt, sitemap.xml (mod_wsgi)
  30. flask sqlalchemy query with keyword as variable

Leave a Comment