Spark groupByKey alternative

by

groupByKey is fine for the case when we want a “smallish” collection of values per key, as in the question.

TL;DR

The “do not use” warning on groupByKey applies for two general cases:

1) You want to aggregate over the values:

  • DON’T: rdd.groupByKey().mapValues(_.sum)
  • DO: rdd.reduceByKey(_ + _)

In this case, groupByKey will waste resouces materializing a collection while what we want is a single element as answer.

2) You want to group very large collections over low cardinality keys:

  • DON’T: allFacebookUsersRDD.map(user => (user.likesCats, user)).groupByKey()
  • JUST DON’T

In this case, groupByKey will potentially result in an OOM error.

groupByKey materializes a collection with all values for the same key in one executor. As mentioned, it has memory limitations and therefore, other options are better depending on the case.

All the grouping functions, like groupByKey, aggregateByKey and reduceByKey rely on the base: combineByKey and therefore no other alternative will be better for the usecase in the question, they all rely on the same common process.

More Related Contents:

  1. How to find median and quantiles using Spark
  2. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  3. Spark union of multiple RDDs
  4. How to extract an element from a array in pyspark
  5. A list as a key for PySpark’s reduceByKey
  6. PySpark DataFrames – way to enumerate without converting to Pandas?
  7. Reduce a key-value pair into a key-list pair with Apache Spark
  8. Random numbers generation in PySpark
  9. Calling Java/Scala function from a task
  10. Efficient string matching in Apache Spark
  11. Load CSV file with Spark
  12. How to link PyCharm with PySpark?
  13. How to turn off INFO logging in Spark?
  14. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  15. importing pyspark in python shell
  16. collect_list by preserving order based on another variable
  17. Rename nested field in spark dataframe
  18. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  19. How to determine if object is a valid key-value pair in PySpark
  20. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  21. Best way to get the max value in a Spark dataframe column
  22. How to map features from the output of a VectorAssembler back to the column names in Spark ML?
  23. Apache Spark Python Cosine Similarity over DataFrames
  24. GroupBy column and filter rows with maximum value in Pyspark
  25. Spark mllib predicting weird number or NaN
  26. Concatenate two PySpark dataframes
  27. Filtering DataFrame using the length of a column
  28. How to process RDDs using a Python class?
  29. What is the Spark DataFrame method `toPandas` actually doing?
  30. how to add Row id in pySpark dataframes [duplicate]

Leave a Comment