Container is running beyond memory limits

by

You should also properly configure the maximum memory allocations for MapReduce. From this HortonWorks tutorial:

[…]

Each machine in our cluster has 48 GB of RAM. Some of this RAM should be >reserved for Operating System usage. On each node, we’ll assign 40 GB RAM for >YARN to use and keep 8 GB for the Operating System

For our example cluster, we have the minimum RAM for a Container
(yarn.scheduler.minimum-allocation-mb) = 2 GB. We’ll thus assign 4 GB
for Map task Containers, and 8 GB for Reduce tasks Containers.

In mapred-site.xml:

mapreduce.map.memory.mb: 4096

mapreduce.reduce.memory.mb: 8192

Each Container will run JVMs for the Map and Reduce tasks. The JVM
heap size should be set to lower than the Map and Reduce memory
defined above, so that they are within the bounds of the Container
memory allocated by YARN.

In mapred-site.xml:

mapreduce.map.java.opts: -Xmx3072m

mapreduce.reduce.java.opts: -Xmx6144m

The above settings configure the upper limit of the physical RAM that
Map and Reduce tasks will use.

To sum it up:

  1. In YARN, you should use the mapreduce configs, not the mapred ones. EDIT: This comment is not applicable anymore now that you’ve edited your question.
  2. What you are configuring is actually how much you want to request, not what is the max to allocate.
  3. The max limits are configured with the java.opts settings listed above.

Finally, you may want to check this other SO question that describes a similar problem (and solution).

More Related Contents:

  1. How does Hadoop process records split across block boundaries?
  2. merge output files after reduce phase
  3. Setting the number of map tasks and reduce tasks
  4. Is it better to use the mapred or the mapreduce package to create a Hadoop Job?
  5. Chaining multiple MapReduce jobs in Hadoop
  6. When do reduce tasks start in Hadoop?
  7. Spark on yarn concept understanding
  8. How can I access S3/S3n from a local Hadoop 2.6 installation?
  9. How to get the input file name in the mapper in a Hadoop program?
  10. Hadoop speculative task execution
  11. Hive unable to manually set number of reducers
  12. how many mappers and reduces will get created for a partitoned table in hive
  13. hadoop map reduce secondary sorting
  14. Apache Spark: The number of cores vs. the number of executors
  15. Apache Hadoop Yarn – Underutilization of cores
  16. Where are logs in Spark on YARN?
  17. Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)
  18. How does Hadoop perform input splits?
  19. What is the use of grouping comparator in hadoop map reduce
  20. Hadoop input split size vs block size
  21. Default number of reducers
  22. What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
  23. What is Google’s Dremel? How is it different from Mapreduce?
  24. Hadoop WordCount example stuck at map 100% reduce 0%
  25. Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?
  26. hadoop: difference between 0 reducer and identity reducer?
  27. How to transpose/pivot data in hive?
  28. Reading file as single record in hadoop
  29. Spark iterate HDFS directory
  30. Pig Latin: Load multiple files from a date range (part of the directory structure)

Leave a Comment