Apache Spark – Scala – ReduceByKey – with keys repeating up to twice only

by

What (probably) takes the time here is shuffling the data: when you want to group two or more records together, they must reside within the same partition, so Spark has to first shuffle the records so that all records with same key are in a single partition.

Now, even if each key has two records at most, this shuffle will have to take place, unless you can somehow guarantee that each key is already contained in a single partition – for example, if you loaded this RDD from HDFS and you somehow know that each key resides on a single file part to begin with. In that (unlikely) case, you can use mapPartitions to perform the grouping yourself on each partition separately, thus saving the shuffle:

vectors.mapPartitions( 
  iter => iter.toList.groupBy(_._1).map { case (k, list) => (k, list.map(_._2).reduce(_ * _)) }.iterator, 
  preservesPartitioning = true)

None of this is special to the case where the maximum repetition of each key is 2, by the way.

More Related Contents:

  1. How MapReduce work in Apache Spark and Scala?
  2. Write to multiple outputs by key Spark – one Spark job
  3. How does HashPartitioner work?
  4. Spark 2.0 Dataset vs DataFrame
  5. Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects
  6. Automatically and Elegantly flatten DataFrame in Spark SQL
  7. How do I skip a header from CSV files in Spark?
  8. Spark extracting values from a Row
  9. Why accesing DataFrame from UDF results in NullPointerException?
  10. SparkSQL: How to deal with null values in user defined function?
  11. How to aggregate values into collection after groupBy?
  12. How to read records in JSON format from Kafka using Structured Streaming?
  13. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  14. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  15. Scala spark, listbuffer is empty
  16. Spark / Scala: forward fill with last observation
  17. Change nullable property of column in spark dataframe
  18. Reading DataFrame from partitioned parquet file
  19. Perform a typed join in Scala with Spark Datasets
  20. Apache Spark: Get number of records per partition
  21. Left Anti join in Spark?
  22. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  23. DataFrame-ified zipWithIndex
  24. Derive multiple columns from a single column in a Spark DataFrame
  25. How to save a spark DataFrame as csv on disk?
  26. How to explode an array into multiple columns in Spark
  27. Why is join not possible after show operator?
  28. How to create DataFrame from Scala’s List of Iterables?
  29. How to create a Dataset of Maps?
  30. Spark ML VectorAssembler returns strange output

Leave a Comment