(Why) do we need to call cache or persist on a RDD

by

Most RDD operations are lazy. Think of an RDD as a description of a series of operations. An RDD is not data. So this line:

val textFile = sc.textFile("/user/emp.txt")

It does nothing. It creates an RDD that says “we will need to load this file”. The file is not loaded at this point.

RDD operations that require observing the contents of the data cannot be lazy. (These are called actions.) An example is RDD.count — to tell you the number of lines in the file, the file needs to be read. So if you write textFile.count, at this point the file will be read, the lines will be counted, and the count will be returned.

What if you call textFile.count again? The same thing: the file will be read and counted again. Nothing is stored. An RDD is not data.

So what does RDD.cache do? If you add textFile.cache to the above code:

val textFile = sc.textFile("/user/emp.txt")
textFile.cache

It does nothing. RDD.cache is also a lazy operation. The file is still not read. But now the RDD says “read this file and then cache the contents”. If you then run textFile.count the first time, the file will be loaded, cached, and counted. If you call textFile.count a second time, the operation will use the cache. It will just take the data from the cache and count the lines.

The cache behavior depends on the available memory. If the file does not fit in the memory, for example, then textFile.count will fall back to the usual behavior and re-read the file.

More Related Contents:

  1. How does HashPartitioner work?
  2. How to convert rdd object to dataframe in spark
  3. Case class equality in Apache Spark
  4. Spark performance for Scala vs Python
  5. Stackoverflow due to long RDD Lineage
  6. How to find spark RDD/Dataframe size?
  7. Modify collection inside a Spark RDD foreach
  8. Parsing multiline records in Scala
  9. Explanation of fold method of spark RDD
  10. Why does partition parameter of SparkContext.textFile not take effect?
  11. Why does Spark RDD partition has 2GB limit for HDFS?
  12. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  13. How to sort an RDD in Scala Spark?
  14. How to transpose an RDD in Spark
  15. Encoder error while trying to map dataframe row to updated row
  16. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  17. How can I change column types in Spark SQL’s DataFrame?
  18. how to make saveAsTextFile NOT split output into multiple file?
  19. Spark losing println() on stdout
  20. Spark DataFrame: does groupBy after orderBy maintain that order?
  21. Spark UDAF with ArrayType as bufferSchema performance issues
  22. Apache Spark how to append new column from list/array to Spark dataframe
  23. Spark: produce RDD[(X, X)] of all possible combinations from RDD[X]
  24. Encode an ADT / sealed trait hierarchy into Spark DataSet column
  25. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  26. Replace null values in Spark DataFrame
  27. Spark RDD default number of partitions
  28. Calculate Cosine Similarity Spark Dataframe
  29. How to create a Dataset of Maps?
  30. Spark ML VectorAssembler returns strange output

Leave a Comment