Why do I have to explicitly tell Spark what to cache?

by

A subjective list of reasons:

  • in practice caching is rarely needed and is useful mostly for iterative algorithms, breaking long lineages. For example typical ETL pipelines may not require caching at all. Cache most of the RDDs is definitely not the right choice.
  • there is no universal caching strategy. Actual choice depends on available recourses like amount of memory, disks (local, remote, storage service), file system (in-memory, on-disk) and particular application.
  • on-disk persistence is expensive, in memory persistence puts more stress on a JVM and is using the most valuable resource in Spark

  • it is impossible to cache automatically without making assumptions about application semantics. In particular:

    • expected behavior when data source changes. There is no universal answer and in many situations it can be impossible to automatically track changes
    • differentiating between deterministic and non-deterministic transformations and choosing between caching and re-computing
  • comparing Spark caching to OS level caching doesn’t make sense. The main goal of OS caching is to reduce latency. In Spark latency is usually not the most important factor and caching is used for other purposes like consistency, correctness and reducing stress on different parts of the system.
  • if cache doesn’t use off-heap storage than caching introduces additional pressure on the garbage collector. GC cost can be actually higher than a cost of recomputing the data.
  • depending on the data and caching method reading data from cache can be significantly less efficient memory-wise.
  • Caching interferes with more advanced optimizations available in Spark SQL, effectively disabling partition pruning or predicate and projection pushdown.

It is also worth to note that:

  • removing cached data is handled automatically using LRU
  • some data (like intermediate shuffle data) is persisted automatically. I acknowledge that it makes some of the previous arguments at least partially invalid.
  • Spark caching doesn’t affect system level or JVM level mechanisms

More Related Contents:

  1. Spark Dataframe validating column names for parquet writes
  2. How to optimize partitioning when migrating data from JDBC source?
  3. Spark: subtract two DataFrames
  4. TypeError: Column is not iterable – How to iterate over ArrayType()?
  5. How to overwrite the output directory in spark
  6. Is groupByKey ever preferred over reduceByKey
  7. Pyspark : forward fill with last observation for a DataFrame
  8. spark.sql.crossJoin.enabled for Spark 2.x
  9. PySpark: how to resample frequencies
  10. Explode array data into rows in spark [duplicate]
  11. Why does Spark think this is a cross / Cartesian join
  12. Why does sortBy transformation trigger a Spark job?
  13. Spark SQL broadcast hash join
  14. What is the difference between cache and persist?
  15. What is the difference between Apache Spark SQLContext vs HiveContext?
  16. Saving dataframe to local file system results in empty results
  17. Dealing with a large gzipped file in Spark
  18. When are accumulators truly reliable?
  19. What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?
  20. pyspark: Efficiently have partitionBy write to same number of total partitions as original table
  21. Spark lists all leaf node even in partitioned data
  22. AWS EMR – ModuleNotFoundError: No module named ‘pyarrow’
  23. How to assign unique contiguous numbers to elements in a Spark RDD
  24. What is spark.driver.maxResultSize?
  25. Apache Spark: Differences between client and cluster deploy modes
  26. Spark + s3 – error – java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
  27. How to get Kafka offsets for structured query for manual and reliable offset management?
  28. When to use Spark DataFrame/Dataset API and when to use plain RDD?
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment