What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?

by

If you’re running out of memory on the shuffle, try setting spark.sql.shuffle.partitions to 2001.

Spark uses a different data structure for shuffle book-keeping when the number of partitions is greater than 2000:

private[spark] object MapStatus {

  def apply(loc: BlockManagerId, uncompressedSizes: Array[Long]): MapStatus = {
    if (uncompressedSizes.length > 2000) {
      HighlyCompressedMapStatus(loc, uncompressedSizes)
    } else {
      new CompressedMapStatus(loc, uncompressedSizes)
    }
  }
...

I really wish they would let you configure this independently.

By the way, I found this information in a Cloudera slide deck.

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Using a column value as a parameter to a spark DataFrame function
  3. Find maximum row per group in Spark DataFrame
  4. Split Spark Dataframe string column into multiple columns
  5. While writing to hdfs path getting error java.io.IOException: Failed to rename
  6. Multiple Aggregate operations on the same column of a spark dataframe
  7. Partitioning in spark while reading from RDBMS via JDBC
  8. How to control partition size in Spark SQL
  9. Avoid performance impact of a single partition mode in Spark window functions
  10. How to import multiple csv files in a single load?
  11. How to access element of a VectorUDT column in a Spark DataFrame?
  12. How to split a list to multiple columns in Pyspark?
  13. pyspark dataframe filter or include based on list
  14. How to save/insert each DStream into a permanent table
  15. Spark load data and add filename as dataframe column
  16. What is the maximum size for a broadcast object in Spark?
  17. PySpark: how to resample frequencies
  18. How does createOrReplaceTempView work in Spark?
  19. Fill in null with previously known good value with pyspark
  20. PySpark – get row number for each row in a group
  21. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  22. pyspark: count distinct over a window
  23. Using windowing functions in Spark
  24. PySpark: How to fillna values in dataframe for specific columns?
  25. Why spark application fail with “executor.CoarseGrainedExecutorBackend: Driver Disassociated”?
  26. What does “Correlated scalar subqueries must be Aggregated” mean?
  27. How to improve broadcast Join speed with between condition in Spark
  28. How to calculate Median in spark sqlContext for column of data type double
  29. Spark SQL referencing attributes of UDT
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment