How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?

by

Partitioners work by assigning a key to a partition. You would need prior knowledge of the key distribution, or look at all keys, to make such a partitioner. This is why Spark does not provide you with one.

In general you do not need such a partitioner. In fact I cannot come up with a use case where I would need equal-size partitions. What if the number of elements is odd?

Anyway, let us say you have an RDD keyed by sequential Ints, and you know how many in total. Then you could write a custom Partitioner like this:

class ExactPartitioner[V](
    partitions: Int,
    elements: Int)
  extends Partitioner {

  def getPartition(key: Any): Int = {
    val k = key.asInstanceOf[Int]
    // `k` is assumed to go continuously from 0 to elements-1.
    return k * partitions / elements
  }
}

More Related Contents:

  1. Write to multiple outputs by key Spark – one Spark job
  2. Spark – load CSV file as DataFrame?
  3. How to get ID of a map task in Spark?
  4. Apache Spark: Get number of records per partition
  5. Spark Scala list folders in directory
  6. Spark: long delay between jobs
  7. Write single CSV file using spark-csv
  8. (Why) do we need to call cache or persist on a RDD
  9. Automatically and Elegantly flatten DataFrame in Spark SQL
  10. How do I skip a header from CSV files in Spark?
  11. Spark extracting values from a Row
  12. Stratified sampling in Spark
  13. How do I convert csv file to rdd
  14. How to pass -D parameter or environment variable to Spark job?
  15. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  16. Append a column to Data Frame in Apache Spark 1.3
  17. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  18. Processing multiple files as independent RDD’s in parallel
  19. MatchError while accessing vector column in Spark 2.0
  20. How to save DataFrame directly to Hive?
  21. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  22. Spark / Scala: forward fill with last observation
  23. How to convert unix timestamp to date in Spark
  24. Renaming column names of a DataFrame in Spark Scala
  25. What is going wrong with `unionAll` of Spark `DataFrame`?
  26. Exploding nested Struct in Spark dataframe
  27. How to get keys and values from MapType column in SparkSQL DataFrame
  28. Addition of two RDD[mllib.linalg.Vector]’s
  29. How to create a Dataset of Maps?
  30. Spark ML VectorAssembler returns strange output

Leave a Comment