Apache Hadoop Yarn – Underutilization of cores

by

The problem lies not with yarn-site.xml or spark-defaults.conf but actually with the resource calculator that assigns the cores to the executors or in the case of MapReduce jobs, to the Mappers/Reducers.

The default resource calculator i.e org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator uses only memory information for allocating containers and CPU scheduling is not enabled by default. To use both memory as well as the CPU, the resource calculator needs to be changed to org.apache.hadoop.yarn.util.resource.DominantResourceCalculator in the capacity-scheduler.xml file.

Here’s what needs to change.

<property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
</property>

More Related Contents:

  1. Spark on yarn concept understanding
  2. Apache Spark: The number of cores vs. the number of executors
  3. Where are logs in Spark on YARN?
  4. Begenner at spark Big data programming (spark code)
  5. Container is running beyond memory limits
  6. How to open/stream .zip files through Spark?
  7. How to get ID of a map task in Spark?
  8. How can I access S3/S3n from a local Hadoop 2.6 installation?
  9. Pyspark: get list of files/directories on HDFS path
  10. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  11. How to access s3a:// files from Apache Spark?
  12. Is it better to have one large parquet file or lots of smaller parquet files?
  13. Easiest way to install Python dependencies on Spark executor nodes?
  14. Spark iterate HDFS directory
  15. Cannot Read a file from HDFS using Spark
  16. Information about big data and hadoop [closed]
  17. Spark – load CSV file as DataFrame?
  18. Read whole text files from a compression in Spark
  19. Spark: Inconsistent performance number in scaling number of cores
  20. Chaining multiple MapReduce jobs in Hadoop
  21. How do I run graphx with Python / pyspark?
  22. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  23. how many mappers and reduces will get created for a partitoned table in hive
  24. How to load data to hive from HDFS without removing the source file?
  25. Loading Data from a .txt file to Table Stored as ORC in Hive
  26. What is the use of grouping comparator in hadoop map reduce
  27. Difference between Pig and Hive? Why have both? [closed]
  28. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  29. What is Google’s Dremel? How is it different from Mapreduce?
  30. Behavior of the parameter “mapred.min.split.size” in HDFS

Leave a Comment