Number of partitions in RDD and performance in Spark

by

The primary effect would be by specifying too few partitions or far too many partitions.

Too few partitions You will not utilize all of the cores available in the cluster.

Too many partitions There will be excessive overhead in managing many small tasks.

Between the two the first one is far more impactful on performance. Scheduling too many smalls tasks is a relatively small impact at this point for partition counts below 1000. If you have on the order of tens of thousands of partitions then spark gets very slow.

More Related Contents:

  1. Spark functions vs UDF performance?
  2. Apache Spark: map vs mapPartitions?
  3. Spark performance for Scala vs Python
  4. How to find median and quantiles using Spark
  5. How do I split an RDD into two or more RDDs?
  6. DataFrame / Dataset groupBy behaviour/optimization
  7. Spark: Inconsistent performance number in scaling number of cores
  8. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  9. Spark union of multiple RDDs
  10. What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism?
  11. Spark groupByKey alternative
  12. How to extract an element from a array in pyspark
  13. A list as a key for PySpark’s reduceByKey
  14. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  15. PySpark DataFrames – way to enumerate without converting to Pandas?
  16. Reduce a key-value pair into a key-list pair with Apache Spark
  17. Apache spark dealing with case statements
  18. Spark ALS predictAll returns empty
  19. Why is the fold action necessary in Spark?
  20. Random numbers generation in PySpark
  21. What is the best way to set a register to zero in x86 assembly: xor, mov or and?
  22. How to speed up A* algorithm at large spatial scales?
  23. Why is XCHG reg, reg a 3 micro-op instruction on modern Intel architectures?
  24. Why should recursion be preferred over iteration?
  25. How to create a custom Estimator in PySpark
  26. PySpark: match the values of a DataFrame column against another DataFrame column
  27. How to get element by Index in Spark RDD (Java)
  28. Return address prediction stack buffer vs stack-stored return address?
  29. When should we use prefetch?
  30. Efficient sse shuffle mask generation for left-packing byte elements

Leave a Comment