How to optimize shuffle spill in Apache Spark application

by

Learning to performance-tune Spark requires quite a bit of investigation and learning. There are a few good resources including this video. Spark 1.4 has some better diagnostics and visualisation in the interface which can help you.

In summary, you spill when the size of the RDD partitions at the end of the stage exceed the amount of memory available for the shuffle buffer.

You can:

  1. Manually repartition() your prior stage so that you have smaller partitions from input.
  2. Increase the shuffle buffer by increasing the memory in your executor processes (spark.executor.memory)
  3. Increase the shuffle buffer by increasing the fraction of executor memory allocated to it (spark.shuffle.memoryFraction) from the default of 0.2. You need to give back spark.storage.memoryFraction.
  4. Increase the shuffle buffer per thread by reducing the ratio of worker threads (SPARK_WORKER_CORES) to executor memory

If there is an expert listening, I would love to know more about how the memoryFraction settings interact and their reasonable range.

More Related Contents:

  1. DataFrame join optimization – Broadcast Hash Join
  2. Spark Dataframe validating column names for parquet writes
  3. How to save/insert each DStream into a permanent table
  4. Spark using python: How to resolve Stage x contains a task of very large size (xxx KB). The maximum recommended task size is 100 KB
  5. reading json file in pyspark
  6. Does a join of co-partitioned RDDs cause a shuffle in Apache Spark?
  7. Airflow BashOperator return exit code 0 even when task failed and return exit code 1
  8. Unable to fetch the value of Println in apache spark
  9. How to serve a Spark MLlib model?
  10. Avoid performance impact of a single partition mode in Spark window functions
  11. Spark DataFrame: count distinct values of every column
  12. Spark Standalone Number Executors/Cores Control
  13. pyspark dataframe filter or include based on list
  14. How to get word details from TF Vector RDD in Spark ML Lib?
  15. Use collect_list and collect_set in Spark SQL
  16. Apache Spark: setting executor instances does not change the executors
  17. How to find mean of grouped Vector columns in Spark SQL?
  18. Generate a Spark StructType / Schema from a case class
  19. Fill in null with previously known good value with pyspark
  20. Keep only duplicates from a DataFrame regarding some field
  21. Which cluster type should I choose for Spark? [closed]
  22. Overwrite only some partitions in a partitioned spark Dataset
  23. Using windowing functions in Spark
  24. What do the numbers on the progress bar mean in spark-shell?
  25. What are broadcast variables? What problems do they solve?
  26. Why spark application fail with “executor.CoarseGrainedExecutorBackend: Driver Disassociated”?
  27. What does “Correlated scalar subqueries must be Aggregated” mean?
  28. Why is the fold action necessary in Spark?
  29. How to add a SparkListener from pySpark in Python?
  30. What will spark do if I don’t have enough memory?

Leave a Comment