Spark Standalone Number Executors/Cores Control

by

Disclaimer: I really don’t know if --num-executors should work or not in standalone mode. I haven’t seen it used outside YARN.

Note: As pointed out by Marco --num-executors is no longer in use on YARN.

You can effectively control number of executors in standalone mode with static allocation (this works on Mesos as well) by combining spark.cores.max and spark.executor.cores where number of executors is determined as:

floor(spark.cores.max / spark.executor.cores)

For example:

--conf "spark.cores.max=4" --conf "spark.executor.cores=2"

More Related Contents:

  1. Which cluster type should I choose for Spark? [closed]
  2. Apache Spark: Differences between client and cluster deploy modes
  3. How to stop INFO messages displaying on spark console?
  4. DataFrame join optimization – Broadcast Hash Join
  5. Overwrite specific partitions in spark dataframe write method
  6. Convert date from String to Date format in Dataframes
  7. What is the meaning of partitionColumn, lowerBound, upperBound, numPartitions parameters?
  8. Spark Transformation – Why is it lazy and what is the advantage?
  9. What is the difference between spark checkpoint and persist to a disk
  10. Xml processing in Spark
  11. What are workers, executors, cores in Spark Standalone cluster?
  12. How to connect to remote hive server from spark [duplicate]
  13. How to fix ‘TypeError: an integer is required (got type bytes)’ error when trying to run pyspark after installing spark 2.4.4
  14. Pyspark : forward fill with last observation for a DataFrame
  15. Adding a group count column to a PySpark dataframe
  16. Multiple Spark applications with HiveContext
  17. NoClassDefFoundError com.apache.hadoop.fs.FSDataInputStream when execute spark-shell
  18. Spark: Best practice for retrieving big data from RDD to local machine
  19. Spark-Obtaining file name in RDDs
  20. How do I add an persistent column of row ids to Spark DataFrame?
  21. How does Spark aggregate function – aggregateByKey work?
  22. Rename more than one column using withColumnRenamed
  23. Convert null values to empty array in Spark DataFrame
  24. DataFrame partitionBy to a single Parquet file (per partition)
  25. Temp table caching with spark-sql
  26. PySpark: How to fillna values in dataframe for specific columns?
  27. How to optimize shuffle spill in Apache Spark application
  28. How to calculate Median in spark sqlContext for column of data type double
  29. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’
  30. Understanding Spark’s caching

Leave a Comment