How to run Airflow on Windows

by

Three Basic Options

I went through a few iterations of this problem and documented them as I went along. The three things I tried were:

  1. Install Airflow directly into Windows 10 – This attempt failed.
  2. Install Airflow into Windows 10 WSL with Ubuntu – This worked great. Note that WSL is Windows Subsystem for Linux, which you can get for free in the Windows store.
  3. Install Airflow into Windows 10 via Docker + Centos – This worked great as well.

Note that if you want to get it running as a Linux service, it is not possible for option number 2. It is possible for option number 3, but I didn’t do it as it requires activating privileged containers in docker (which I wan’t aware of when I started). Also, running a service in Docker is kind of against paradigm as each container should be a single process/unit of responsibility anyway.

Detailed Description of #2 – WSL Option

If you’re gong for option 2, the basic steps are:

  • Get WSL Ubuntu installed and opened up.
  • Verify it comes with python 3.6.5 or so (python3 -version).
  • Assuming it still does, add these packages so that installing PIP will work.
    • sudo apt-get install software-properties-common
    • sudo apt-add-repository universe
    • sudo apt-get update
  • Install pip with:
    • sudo apt-get install python-pip (or python3-pip for Python 3)
  • Run the following 2 commands to install airflow:
    • export SLUGIFY_USES_TEXT_UNIDECODE=yes
    • pip install apache-airflow (or pip3 for Python 3)
  • Open a new terminal (I was surprised, but this seemed to be required).
  • Init the airflow DB:
    • airflow initdb

After this, you should be good to go! The blog has more detail on many of these steps and rough timelines for how long setting up WSL takes, etc – so if you have a hard time dive in there some more.

More Related Contents:

  1. Flask CLI throws ‘OSError: [Errno 8] Exec format error’ when run through docker-compose
  2. Tyring to set up modelview with Flask-Admin causes ImportError
  3. How to return a dict as a JSON response from a Flask view?
  4. How to get POSTed JSON in Flask?
  5. “python” not recognized as a command
  6. Windows is not passing command line arguments to Python programs executed from the shell
  7. Python pysftp get_r from Linux works fine on Linux but not on Windows
  8. Which is the easiest way to simulate keyboard and mouse on Python?
  9. fcntl substitute on Windows
  10. Python Window Activation
  11. Python command not working in command prompt
  12. Using Python, how can I access a shared folder on windows network?
  13. Handle Flask requests concurrently with threaded=True
  14. Saving upload in Flask only saves to project root
  15. What is the purpose of Flask’s context stacks?
  16. Python running as Windows Service: OSError: [WinError 6] The handle is invalid
  17. tkinter.TclError: image “pyimage3” doesn’t exist
  18. How do I install SciPy on 64 bit Windows?
  19. pyclipper installation error: “tp_print is not a member of _typeobject”
  20. Passing a matplotlib figure to HTML (flask)
  21. How do I set response headers in Flask?
  22. Restrict static file access to logged in users
  23. Flask sqlalchemy many-to-many insert data
  24. How to stop Python closing immediately when executed in Microsoft Windows
  25. Can’t import dll module in Python
  26. Flask(‘application’) versus Flask(__name__)
  27. Why python executable opens new window instance when function by multiprocessing module is called on windows
  28. Flask-SQLAlchemy Constructor
  29. Running Scapy on Windows with Python 2.7
  30. How can I stop Python’s csv.DictWriter.writerows from adding empty lines between rows in Windows?

Leave a Comment