Spark Transformation – Why is it lazy and what is the advantage?

by

For transformations, Spark adds them to a DAG of computation and only when driver requests some data, does this DAG actually gets executed.

One advantage of this is that Spark can make many optimization decisions after it had a chance to look at the DAG in entirety. This would not be possible if it executed everything as soon as it got it.

For example — if you executed every transformation eagerly, what does that mean? Well, it means you will have to materialize that many intermediate datasets in memory. This is evidently not efficient — for one, it will increase your GC costs. (Because you’re really not interested in those intermediate results as such. Those are just convnient abstractions for you while writing the program.) So, what you do instead is — you tell Spark what is the eventual answer you’re interested and it figures out best way to get there.

More Related Contents:

  1. Using a column value as a parameter to a spark DataFrame function
  2. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  3. Split Spark Dataframe string column into multiple columns
  4. What does “Stage Skipped” mean in Apache Spark web UI?
  5. How to control partition size in Spark SQL
  6. How to import multiple csv files in a single load?
  7. How are stages split into tasks in Spark?
  8. How to tune spark executor number, cores and executor memory?
  9. Which operations preserve RDD order?
  10. Spark Standalone Number Executors/Cores Control
  11. pyspark dataframe filter or include based on list
  12. How to get word details from TF Vector RDD in Spark ML Lib?
  13. Use collect_list and collect_set in Spark SQL
  14. Apache Spark: setting executor instances does not change the executors
  15. How to find mean of grouped Vector columns in Spark SQL?
  16. Generate a Spark StructType / Schema from a case class
  17. Fill in null with previously known good value with pyspark
  18. How to group by common element in array?
  19. Spark using python: How to resolve Stage x contains a task of very large size (xxx KB). The maximum recommended task size is 100 KB
  20. Keep only duplicates from a DataFrame regarding some field
  21. reading json file in pyspark
  22. Which cluster type should I choose for Spark? [closed]
  23. Overwrite only some partitions in a partitioned spark Dataset
  24. Using windowing functions in Spark
  25. reduce result datasets into single dataset
  26. How to exclude multiple columns in Spark dataframe in Python
  27. Why does Spark job fail with “too many open files”?
  28. How does Distinct() function work in Spark?
  29. Stratified sampling with pyspark
  30. Get CSV to Spark dataframe

Leave a Comment