Explanation of fold method of spark RDD

by

Well, it is actually pretty well explained by the official documentation:

Aggregate the elements of each partition, and then the results for all the partitions, using a given associative and commutative function and a neutral “zero value”. The function op(t1, t2) is allowed to modify t1 and return it as its result value to avoid object allocation; however, it should not modify t2.

This behaves somewhat differently from fold operations implemented for non-distributed collections in functional languages like Scala. This fold operation may be applied to partitions individually, and then fold those results into the final result, rather than apply the fold to each element sequentially in some defined ordering. For functions that are not commutative, the result may differ from that of a fold applied to a non-distributed collection.

To illustrate what is going on lets try to simulate what is going on step by step:

val rdd = sc.parallelize(Array(2., 3.))

val byPartition = rdd.mapPartitions(
    iter => Array(iter.fold(0.0)((p, v) => (p +  v * v))).toIterator).collect()

It gives us something similar to this Array[Double] = Array(0.0, 0.0, 0.0, 4.0, 0.0, 0.0, 0.0, 9.0) and

byPartition.reduce((p, v) => (p + v * v))

returns 97

Important thing to note is that results can differ from run to run depending on an order in which partitions are combined.

More Related Contents:

  1. How does HashPartitioner work?
  2. How to convert rdd object to dataframe in spark
  3. (Why) do we need to call cache or persist on a RDD
  4. Case class equality in Apache Spark
  5. Spark performance for Scala vs Python
  6. Stackoverflow due to long RDD Lineage
  7. How to find spark RDD/Dataframe size?
  8. Modify collection inside a Spark RDD foreach
  9. Parsing multiline records in Scala
  10. Why does partition parameter of SparkContext.textFile not take effect?
  11. Why does Spark RDD partition has 2GB limit for HDFS?
  12. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  13. How to sort an RDD in Scala Spark?
  14. How to transpose an RDD in Spark
  15. How to store custom objects in Dataset?
  16. Split 1 column into 3 columns in spark scala
  17. Provide schema while reading csv file as a dataframe
  18. How to flatmap a nested Dataframe in Spark
  19. Joining Spark dataframes on the key
  20. Matrix Multiplication in Apache Spark [closed]
  21. How to write unit tests in Spark 2.0+?
  22. Customize SparkContext using sparkConf.set(..) when using spark-shell
  23. Replace missing values with mean – Spark Dataframe
  24. How to calculate the size of dataframe in bytes in Spark?
  25. Spark Scala list folders in directory
  26. Exception in thread “main” java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)
  27. How to remove parentheses around records when saveAsTextFile on RDD[(String, Int)]?
  28. How can I connect to a postgreSQL database into Apache Spark using scala?
  29. Spark column string replace when present in other column (row)
  30. How to vectorize DataFrame columns for ML algorithms?

Leave a Comment