Spark – repartition() vs coalesce()

by

It avoids a full shuffle. If it’s known that the number is decreasing then the executor can safely keep data on the minimum number of partitions, only moving the data off the extra nodes, onto the nodes that we kept.

So, it would go something like this:

Node 1 = 1,2,3
Node 2 = 4,5,6
Node 3 = 7,8,9
Node 4 = 10,11,12

Then coalesce down to 2 partitions:

Node 1 = 1,2,3 + (10,11,12)
Node 3 = 7,8,9 + (4,5,6)

Notice that Node 1 and Node 3 did not require its original data to move.

More Related Contents:

  1. What is the difference between cache and persist?
  2. Unable to fetch the value of Println in apache spark
  3. How do I split an RDD into two or more RDDs?
  4. What does “Stage Skipped” mean in Apache Spark web UI?
  5. Spark: subtract two DataFrames
  6. Which operations preserve RDD order?
  7. Is groupByKey ever preferred over reduceByKey
  8. What are workers, executors, cores in Spark Standalone cluster?
  9. Default Partitioning Scheme in Spark
  10. How DAG works under the covers in RDD?
  11. Spark parquet partitioning : Large number of files
  12. What is a task in Spark? How does the Spark worker execute the jar file?
  13. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  14. Why does sortBy transformation trigger a Spark job?
  15. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  16. Apache spark dealing with case statements
  17. How does Spark aggregate function – aggregateByKey work?
  18. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  19. What is spark.driver.maxResultSize?
  20. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  21. Spark ALS predictAll returns empty
  22. Why is the fold action necessary in Spark?
  23. Spark Standalone Number Executors/Cores Control
  24. How to get word details from TF Vector RDD in Spark ML Lib?
  25. Apache Spark: setting executor instances does not change the executors
  26. Number of partitions in RDD and performance in Spark
  27. Keep only duplicates from a DataFrame regarding some field
  28. Overwrite only some partitions in a partitioned spark Dataset
  29. Using windowing functions in Spark
  30. PySpark: How to fillna values in dataframe for specific columns?

Leave a Comment