For Apache Airflow, How can I pass the parameters when manually trigger DAG via CLI?

You can pass parameters from the CLI using –conf ‘{“key”:”value”}’ and then use it in the DAG file as “{{ dag_run.conf[“key”] }}” in templated field. CLI: airflow trigger_dag ‘example_dag_conf’ -r ‘run_id’ –conf ‘{“message”:”value”}’ DAG File: args = { ‘start_date’: datetime.utcnow(), ‘owner’: ‘airflow’, } dag = DAG( dag_id=’example_dag_conf’, default_args=args, schedule_interval=None, ) def run_this_func(ds, **kwargs): print(“Remotely received … Read more

Airflow: how to delete a DAG?

Edit 8/27/18 – Airflow 1.10 is now released on PyPI! https://pypi.org/project/apache-airflow/1.10.0/ How to delete a DAG completely We have this feature now in Airflow ≥ 1.10! The PR #2199 (Jira: AIRFLOW-1002) adding DAG removal to Airflow has now been merged which allows fully deleting a DAG’s entries from all of the related tables. The core … Read more

Wiring top-level DAGs together

Taking hints from @Viraj Parekh‘s answer, I was able to make TriggerDagRunOperator work in the intended fashion. I’m hereby posting my (partial) answer; will update as and when things become clear. How to overcome limitation of parent_id prefix in dag_id of SubDags? As told @Viraj, there’s no straight way of achieving this. Extending SubDagOperator to … Read more