Apache Spark: setting executor instances does not change the executors

by

Increase yarn.nodemanager.resource.memory-mb in yarn-site.xml

With 12g per node you can only launch driver(3g) and 2 executors(11g).

Node1 – driver 3g (+7% overhead)

Node2 – executor1 11g (+7% overhead)

Node3 – executor2 11g (+7% overhead)

now you are requesting for executor3 of 11g and no node has 11g memory available.

for 7% overhead refer spark.yarn.executor.memoryOverhead and spark.yarn.driver.memoryOverhead in https://spark.apache.org/docs/1.2.0/running-on-yarn.html

More Related Contents:

  1. Spark yarn cluster vs client – how to choose which one to use?
  2. Which cluster type should I choose for Spark? [closed]
  3. Using a column value as a parameter to a spark DataFrame function
  4. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  5. Split Spark Dataframe string column into multiple columns
  6. What does “Stage Skipped” mean in Apache Spark web UI?
  7. How to control partition size in Spark SQL
  8. How to import multiple csv files in a single load?
  9. How are stages split into tasks in Spark?
  10. How to tune spark executor number, cores and executor memory?
  11. Which operations preserve RDD order?
  12. How to split a list to multiple columns in Pyspark?
  13. Apache Spark does not delete temporary directories
  14. How to manually set group.id and commit kafka offsets in spark structured streaming?
  15. Spark parquet partitioning : Large number of files
  16. How to find mean of grouped Vector columns in Spark SQL?
  17. Generate a Spark StructType / Schema from a case class
  18. Fill in null with previously known good value with pyspark
  19. Spark using python: How to resolve Stage x contains a task of very large size (xxx KB). The maximum recommended task size is 100 KB
  20. Keep only duplicates from a DataFrame regarding some field
  21. reading json file in pyspark
  22. Overwrite only some partitions in a partitioned spark Dataset
  23. Using windowing functions in Spark
  24. How to assign unique contiguous numbers to elements in a Spark RDD
  25. What is spark.driver.maxResultSize?
  26. Apache Spark: Differences between client and cluster deploy modes
  27. Spark + s3 – error – java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
  28. How to get Kafka offsets for structured query for manual and reliable offset management?
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment