Default Partitioning Scheme in Spark

by

You have to distinguish between two different things:

  • partitioning as distributing data between partitions depending on a value of the key which is limited only to the PairwiseRDDs (RDD[(T, U)]). This creates a relationship between partition and the set of keys which can be found on a given partition.

  • partitioning as splitting input into multiple partitions where data is simply divided into chunks containing consecutive records to enable distributed computation. Exact logic depends on a specific source but it is either number of records or size of a chunk.

    In case of parallelize data is evenly distributed between partitions using indices. In case of HadoopInputFormats (like textFile) it depends on properties like mapreduce.input.fileinputformat.split.minsize / mapreduce.input.fileinputformat.split.maxsize.

So default partitioning scheme is simply none because partitioning is not applicable to all RDDs. For operations which require partitioning on a PairwiseRDD (aggregateByKey, reduceByKey etc.) default method is use hash partitioning.

More Related Contents:

  1. Why does sortBy transformation trigger a Spark job?
  2. Unable to fetch the value of Println in apache spark
  3. Spark – repartition() vs coalesce()
  4. How does HashPartitioner work?
  5. How do I split an RDD into two or more RDDs?
  6. Partitioning in spark while reading from RDBMS via JDBC
  7. What does “Stage Skipped” mean in Apache Spark web UI?
  8. How to control partition size in Spark SQL
  9. Avoid performance impact of a single partition mode in Spark window functions
  10. How to optimize partitioning when migrating data from JDBC source?
  11. Spark: subtract two DataFrames
  12. Which operations preserve RDD order?
  13. Is groupByKey ever preferred over reduceByKey
  14. How DAG works under the covers in RDD?
  15. Spark parquet partitioning : Large number of files
  16. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  17. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  18. Apache spark dealing with case statements
  19. What is the difference between cache and persist?
  20. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  21. Spark lists all leaf node even in partitioned data
  22. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  23. Spark ALS predictAll returns empty
  24. Why is the fold action necessary in Spark?
  25. How to stop INFO messages displaying on spark console?
  26. DataFrame join optimization – Broadcast Hash Join
  27. What is the meaning of partitionColumn, lowerBound, upperBound, numPartitions parameters?
  28. What is the difference between spark checkpoint and persist to a disk
  29. How to exclude multiple columns in Spark dataframe in Python
  30. Why does Spark job fail with “too many open files”?

Leave a Comment