What is the difference between cache and persist?

by

With cache(), you use only the default storage level :

  • MEMORY_ONLY for RDD
  • MEMORY_AND_DISK for Dataset

With persist(), you can specify which storage level you want for both RDD and Dataset.

From the official docs:

  • You can mark an RDD to be persisted using the persist() or cache() methods on it.
  • each persisted RDD can be stored using a different storage level
  • The cache() method is a shorthand for using the default storage level, which is StorageLevel.MEMORY_ONLY (store deserialized objects in memory).

Use persist() if you want to assign a storage level other than :

  • MEMORY_ONLY to the RDD
  • or MEMORY_AND_DISK for Dataset

Interesting link for the official documentation : which storage level to choose

More Related Contents:

  1. Spark – repartition() vs coalesce()
  2. Unable to fetch the value of Println in apache spark
  3. How do I split an RDD into two or more RDDs?
  4. What does “Stage Skipped” mean in Apache Spark web UI?
  5. Spark: subtract two DataFrames
  6. Which operations preserve RDD order?
  7. Is groupByKey ever preferred over reduceByKey
  8. What are workers, executors, cores in Spark Standalone cluster?
  9. Default Partitioning Scheme in Spark
  10. How DAG works under the covers in RDD?
  11. Spark parquet partitioning : Large number of files
  12. What is a task in Spark? How does the Spark worker execute the jar file?
  13. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  14. Why does sortBy transformation trigger a Spark job?
  15. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  16. Apache spark dealing with case statements
  17. How does Spark aggregate function – aggregateByKey work?
  18. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  19. What is spark.driver.maxResultSize?
  20. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  21. Spark ALS predictAll returns empty
  22. Why is the fold action necessary in Spark?
  23. Difference between DataFrame, Dataset, and RDD in Spark
  24. Apache Spark does not delete temporary directories
  25. Efficient pyspark join
  26. ALS model – how to generate full_u * v^t * v?
  27. PySpark: How to fillna values in dataframe for specific columns?
  28. How to calculate Median in spark sqlContext for column of data type double
  29. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’
  30. Understanding Spark’s caching

Leave a Comment