You are looking for all paths between one node and another in a directed acyclic graph (DAG). A tree is always a DAG, but a DAG isn’t always a tree. The difference is that a tree’s branches are not allowed to join, only divide, while a DAG’s branches can flow together, so long as no … Read more
when using nested withColumns and window functions? Let’s say I want to do: w1 = …rangeBetween(-300, 0) w2 = …rowsBetween(-1,0) (df.withColumn(“some1”, col(f.max(“original1”).over(w1)) .withColumn(“some2”, lag(“some1”)).over(w2)).show() I got a lot of memory problems and high spill even with very small datasets. If I do the same using select instead of withColumn it performs way faster. df.select( f.max(col(“original1”)).over(w1).alias(“some1”), … Read more
Your issue is the start_date being set for the current time. Airflow runs jobs at the end of an interval, not the beginning. This means that the first run of your job is going to be after the first interval. Example: You make a dag and put it live in Airflow at midnight. Today (20XX-01-01 … Read more
pass task instance variables as op_args
I would try to sort the graph topologically, and if you can’t, then it has cycles.
Even i have been looking in the web to learn about how spark computes the DAG from the RDD and subsequently executes the task. At high level, when any action is called on the RDD, Spark creates the DAG and submits it to the DAG scheduler. The DAG scheduler divides operators into stages of tasks. … Read more
The other answer works fine but is too complicated. A simpler way is to just execute Kahn’s algorithm but in parallel. The key is to execute all the tasks in parallel for whom all dependencies have been executed. import java.time.Instant; import java.util.ArrayList; import java.util.List; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.atomic.AtomicInteger; class DependencyManager { … Read more