Spark DAG differs with ‘withColumn’ vs ‘select’

by

when using nested withColumns and window functions?

Let’s say I want to do:

w1 = ...rangeBetween(-300, 0)
w2 = ...rowsBetween(-1,0)

(df.withColumn("some1", col(f.max("original1").over(w1))
   .withColumn("some2", lag("some1")).over(w2)).show()

I got a lot of memory problems and high spill even with very small datasets. If I do the same using select instead of withColumn it performs way faster.

df.select(
    f.max(col("original1")).over(w1).alias("some1"),
    f.lag("some1")).over(w2)
).show()

More Related Contents:

  1. How to add a constant column in a Spark DataFrame?
  2. Spark Dataframe distinguish columns with duplicated name
  3. Pyspark: Split multiple array columns into rows
  4. How do I add a new column to a Spark DataFrame (using PySpark)?
  5. How to change a dataframe column from String type to Double type in PySpark?
  6. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  7. Retrieve top n in each group of a DataFrame in pyspark
  8. Rename nested field in spark dataframe
  9. Filtering a Pyspark DataFrame with SQL-like IN clause
  10. Updating a dataframe column in spark
  11. Dividing complex rows of dataframe to simple rows in Pyspark
  12. Filter Pyspark dataframe column with None value
  13. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  14. How to explode multiple columns of a dataframe in pyspark
  15. Spark add new column to dataframe with value from previous row
  16. PySpark: multiple conditions in when clause
  17. Pyspark: Replacing value in a column by searching a dictionary
  18. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  19. Create Spark DataFrame. Can not infer schema for type
  20. Pivot String column on Pyspark Dataframe
  21. Filtering DataFrame using the length of a column
  22. Multiple condition filter on dataframe
  23. How to return a “Tuple type” in a UDF in PySpark?
  24. How to split Vector into columns – using PySpark
  25. creating spark data structure from multiline record
  26. Pyspark: explode json in column to multiple columns
  27. Cast column containing multiple string date formats to DateTime in Spark
  28. Calculating the cosine similarity between all the rows of a dataframe in pyspark
  29. Build a hierarchy from a relational data-set using Pyspark
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment