Spark: produce RDD[(X, X)] of all possible combinations from RDD[X]

by

Cartesian product and combinations are two different things, the cartesian product will create an RDD of size rdd.size() ^ 2 and combinations will create an RDD of size rdd.size() choose 2 

val rdd = sc.parallelize(1 to 5)
val combinations = rdd.cartesian(rdd).filter{ case (a,b) => a < b }`.
combinations.collect()

Note this will only work if an ordering is defined on the elements of the list, since we use <. This one only works for choosing two but can easily be extended by making sure the relationship a < b for all a and b in the sequence

More Related Contents:

  1. (Why) do we need to call cache or persist on a RDD
  2. Automatically and Elegantly flatten DataFrame in Spark SQL
  3. How do I skip a header from CSV files in Spark?
  4. Spark extracting values from a Row
  5. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  6. How to pass -D parameter or environment variable to Spark job?
  7. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  8. Append a column to Data Frame in Apache Spark 1.3
  9. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  10. Processing multiple files as independent RDD’s in parallel
  11. MatchError while accessing vector column in Spark 2.0
  12. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  13. Spark / Scala: forward fill with last observation
  14. About how to add a new column to an existing DataFrame with random values in Scala
  15. How to convert unix timestamp to date in Spark
  16. Modify collection inside a Spark RDD foreach
  17. Function returns an empty List in Spark
  18. How to write unit tests in Spark 2.0+?
  19. How to compare two dataframe and print columns that are different in scala
  20. Operate on neighbor elements in RDD in Spark
  21. Customize SparkContext using sparkConf.set(..) when using spark-shell
  22. Renaming column names of a DataFrame in Spark Scala
  23. What is going wrong with `unionAll` of Spark `DataFrame`?
  24. Spark: Transpose DataFrame Without Aggregating
  25. Spark: long delay between jobs
  26. How to get keys and values from MapType column in SparkSQL DataFrame
  27. Addition of two RDD[mllib.linalg.Vector]’s
  28. How to print the contents of RDD?
  29. Scala-Spark Dynamically call groupby and agg with parameter values
  30. Joining two dataframes without a common column

Leave a Comment