Airflow: how to get pip packages installed via their docker-compose.yml?

by

There is a pretty detailed guide on how to achieve what you are looking for on the Airflow docs here. Depending on your requirements, this may be as easy as extending the original image using a From directive while creating a new Dockerfile, or you may need to customize the image to suit your needs.

If you go with the Extending the image approach your new Dockerfile will be something like this:

FROM apache/airflow:2.0.1
USER root
RUN apt-get update \
  && apt-get install -y --no-install-recommends \
         build-essential my-awesome-apt-dependency-to-add \
  && apt-get autoremove -yqq --purge \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*
USER airflow
RUN pip install --no-cache-dir --user my-awesome-pip-dependency-to-add

Then you could just add something like these to the docker-compose file:

...
version: "3"
x-airflow-common: &airflow-common
  build: . # this is optional
  image: ${AIRFLOW_IMAGE_NAME:-the_name_of_your_extended_image
  ...
...

Finally, build your image and turn everything back on using compose. Try the docs for details or a full explanation. Hope that works for you!

More Related Contents:

  1. How to copy files from host to Docker container?
  2. Why docker container exits immediately
  3. Add a volume to Docker, but exclude a sub-folder
  4. How to change the docker image installation directory?
  5. Locating data volumes in Docker Desktop (Windows)
  6. Docker Compose – Share named volume between multiple containers
  7. How to keep Docker container running after starting services?
  8. Docker remove TAG images
  9. what is docker run -it flag?
  10. Cannot login to Docker account
  11. Am I trying to connect to a TLS-enabled daemon without TLS?
  12. Keras with TensorFlow backend not using GPU
  13. Docker: Add a restart policy to a container that was already created
  14. How to set a path on host for a named volume in docker-compose.yml
  15. How can I use a local file on container?
  16. Port forwarding in docker-machine?
  17. Docker expose all ports or range of ports from 7000 to 8000
  18. write in shared volumes docker
  19. How to port data-only volumes from one host to another?
  20. What is the alternative to condition form of depends_on in docker-compose Version 3?
  21. How to execute a script when I terminate a docker container
  22. Add Insecure Registry to Docker
  23. How to get remote access to a private docker-registry?
  24. Installing numpy on Docker Alpine
  25. How to add initial users when starting a RabbitMQ Docker container?
  26. Connect to another container using Docker compose
  27. How to share localhost between two different Docker containers?
  28. How many CPUs does a docker container use?
  29. What is the authoritative list of Docker Run exit codes?
  30. How to stop all containers when one container stops with docker-compose?

Leave a Comment