Determining optimal number of Spark partitions based on workers, cores and DataFrame size

by

Yes, a spark application has one and only Driver.

What is the relationship between numWorkerNodes and numExecutors?

A worker can host multiple executors, you can think of it like the worker to be the machine/node of your cluster and the executor to be a process (executing in a core) that runs on that worker.

So `numWorkerNodes <= numExecutors’.

Is there any ration for them?

Personally, having worked in a fake cluster, where my laptop was the Driver and a virtual machine in the very same laptop was the worker, and in an industrial cluster of >10k nodes, I didn’t need to care about that, since it seems that spark takes care of that.

I just use:

--num-executors 64

when I launch/submit my script and spark knows, I guess, how many workers it needs to summon (of course, by taking into account other parameters as well, and the nature of the machines).

Thus, personally, I don’t know any such ratio.

Is there a known/generally-accepted/optimal ratio of numDFRows to numPartitions?

I am not aware of one, but as a rule of thumb you could rely on the product of #executors by #executor.cores, and then multiply that by 3 or 4. Of course this is a heuristic. In pyspark it would look like this:

sc = SparkContext(appName = "smeeb-App")
total_cores = int(sc._conf.get('spark.executor.instances')) * int(sc._conf.get('spark.executor.cores'))
dataset = sc.textFile(input_path, total_cores * 3)

How does one calculate the ‘optimal’ number of partitions based on the size of the DataFrame?

That’s a great question. Of course its hard to answer and it depends on your data, cluster, etc., but as discussed here with myself.

Too few partitions and you will have enormous chunks of data, especially when you are dealing with bigdata, thus putting your application in memory stress.

Too many partitions and you will have your hdfs taking much pressure, since all the metadata that has to be generated from the hdfs increases significantly as the number of partitions increase (since it maintains temp files, etc.). *

So what you want is too find a sweet spot for the number of partitions, which is one of the parts of fine tuning your application. 🙂

‘rule of thumb’ is: numPartitions = numWorkerNodes * numCpuCoresPerWorker, is it true?

Ah, I was writing the heuristic above before seeing this. So this is already answered, but take into account the difference of a worker and an executor.

* I just failed for this today: Prepare my bigdata with Spark via Python, when using too many partitions caused Active tasks is a negative number in Spark UI.

More Related Contents:

  1. Spark parquet partitioning : Large number of files
  2. Finding duplicates from large data set using Apache Spark
  3. Spark – repartition() vs coalesce()
  4. Partitioning in spark while reading from RDBMS via JDBC
  5. Overwrite specific partitions in spark dataframe write method
  6. How to control partition size in Spark SQL
  7. Avoid performance impact of a single partition mode in Spark window functions
  8. How to import multiple csv files in a single load?
  9. How to optimize partitioning when migrating data from JDBC source?
  10. How to improve performance for slow Spark jobs using DataFrame and JDBC connection?
  11. TypeError: Column is not iterable – How to iterate over ArrayType()?
  12. What are workers, executors, cores in Spark Standalone cluster?
  13. Python Spark Cumulative Sum by Group Using DataFrame
  14. How to save/insert each DStream into a permanent table
  15. Default Partitioning Scheme in Spark
  16. What is a task in Spark? How does the Spark worker execute the jar file?
  17. Why does sortBy transformation trigger a Spark job?
  18. Pyspark: Pass multiple columns in UDF
  19. Apache spark dealing with case statements
  20. How does Spark aggregate function – aggregateByKey work?
  21. What is the difference between cache and persist?
  22. Spark lists all leaf node even in partitioned data
  23. What is spark.driver.maxResultSize?
  24. Spark: Reading files using different delimiter than new line
  25. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  26. How to loop through each row of dataFrame in pyspark
  27. How to group by common element in array?
  28. How do I add an persistent column of row ids to Spark DataFrame?
  29. Convert null values to empty array in Spark DataFrame
  30. Spark ALS predictAll returns empty

Leave a Comment