Understanding Spark’s caching

by

It would seem that Option B is required. The reason is related to how persist/cache and unpersist are executed by Spark. Since RDD transformations merely build DAG descriptions without execution, in Option A by the time you call unpersist, you still only have job descriptions and not a running execution.

This is relevant because a cache or persist call just adds the RDD to a Map of RDDs that marked themselves to be persisted during job execution. However, unpersist directly tells the blockManager to evict the RDD from storage and removes the reference in the Map of persistent RDDs.

persist function

unpersist function

So you would need to call unpersist after Spark actually executed and stored the RDD with the block manager.

The comments for the RDD.persist method hint towards this:
rdd.persist

More Related Contents:

  1. How to stop INFO messages displaying on spark console?
  2. DataFrame join optimization – Broadcast Hash Join
  3. Overwrite specific partitions in spark dataframe write method
  4. Convert date from String to Date format in Dataframes
  5. What is the meaning of partitionColumn, lowerBound, upperBound, numPartitions parameters?
  6. Spark Transformation – Why is it lazy and what is the advantage?
  7. What is the difference between spark checkpoint and persist to a disk
  8. Xml processing in Spark
  9. What are workers, executors, cores in Spark Standalone cluster?
  10. Spark: Reading files using different delimiter than new line
  11. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  12. Adding a group count column to a PySpark dataframe
  13. Why does format(“kafka”) fail with “Failed to find data source: kafka.” (even with uber-jar)?
  14. How to loop through each row of dataFrame in pyspark
  15. NoClassDefFoundError com.apache.hadoop.fs.FSDataInputStream when execute spark-shell
  16. How to group by common element in array?
  17. Spark: Best practice for retrieving big data from RDD to local machine
  18. Understanding Spark serialization
  19. Pyspark: Pass multiple columns in UDF
  20. Spark-Obtaining file name in RDDs
  21. How does Spark aggregate function – aggregateByKey work?
  22. Convert null values to empty array in Spark DataFrame
  23. spark-shell error on Windows – can it be ignored if not using hadoop?
  24. How can I access python variable in Spark SQL?
  25. How to exclude multiple columns in Spark dataframe in Python
  26. Apache Spark: Differences between client and cluster deploy modes
  27. Why does Spark job fail with “too many open files”?
  28. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  29. How to use groupBy to collect rows into a map?
  30. When to use Spark DataFrame/Dataset API and when to use plain RDD?

Leave a Comment