Airflow “This DAG isnt available in the webserver DagBag object “

by

It is not you nor it is correct or expected behavior.
It is a current ‘bug’ with Airflow.
The web server is caching the DagBag in a way that you cannot really use it as expected.

“Attempt removing DagBag caching for the web server” remains on the official TODO as part of the roadmap, indicating that this bug may not yet be fully resolved, but here are some suggestions on how to proceed:

only use builders in airflow v1.9+

Prior to airflow v1.9 this occurs when a dag is instantiated by a function which is imported into the file where instantiation happens. That is: when a builder or factory pattern is used. Some reports of this issue on github 2 and JIRA 3 led to a fix released with in airflow v1.9.

If you are using an older version of airflow, don’t use builder functions.

airflow backfill to reload the cache

As Dmitri suggests, running airflow backfill '<dag_id>' -s '<date>' -e '<date>' for the same start and end date can sometimes help. Thereafter you may end up with the (non)-issue that Priyank points, but that is expected behavior (state: paused or not) depending on the configuration you have in your installation.

More Related Contents:

  1. Proper way to create dynamic workflows in Airflow
  2. Importing an ipynb file from another ipynb file?
  3. setting up s3 for logs in airflow
  4. How to create a conditional task in Airflow
  5. Importing modules in Python – best practice
  6. How to run Airflow on Windows
  7. pass task instance variables as op_args
  8. How to prevent airflow from backfilling dag runs?
  9. How to control the parallelism or concurrency of an Airflow installation?
  10. Apache Airflow scheduler does not trigger DAG at schedule time
  11. How to create a django ViewFlow process programmatically
  12. Import multiple csv files into pandas and concatenate into one DataFrame
  13. Proxies with Python ‘Requests’ module
  14. Is it pythonic: naming lambdas
  15. Break // in x axis of matplotlib [duplicate]
  16. DLL load failed when importing PyQt5
  17. Programmatically searching google in Python using custom search
  18. Convert RGB color to English color name, like ‘green’ with Python
  19. Using a Python Dictionary as a Key (Non-nested)
  20. When to use os.name, sys.platform, or platform.system?
  21. Get the Olson TZ name for the local timezone?
  22. What are the differences between Bot and Client?
  23. Does Python evaluate if’s conditions lazily? [duplicate]
  24. python list comprehension double for
  25. Generating non-repeating random numbers in Python
  26. Matplotlib: Add colorbar to non-mappable object
  27. How to dynamically access class properties in Python?
  28. Making a clock in kivy
  29. Working with floating point NumPy arrays for comparison and related operations
  30. Serve a dynamically generated image with Django

Leave a Comment