Setting the number of map tasks and reduce tasks

by

The number of map tasks for a given job is driven by the number of input splits and not by the mapred.map.tasks parameter. For each input split a map task is spawned. So, over the lifetime of a mapreduce job the number of map tasks is equal to the number of input splits. mapred.map.tasks is just a hint to the InputFormat for the number of maps.

In your example Hadoop has determined there are 24 input splits and will spawn 24 map tasks in total. But, you can control how many map tasks can be executed in parallel by each of the task tracker.

Also, removing a space after -D might solve the problem for reduce.

For more information on the number of map and reduce tasks, please look at the below url

https://cwiki.apache.org/confluence/display/HADOOP2/HowManyMapsAndReduces

More Related Contents:

  1. How does Hadoop process records split across block boundaries?
  2. merge output files after reduce phase
  3. Container is running beyond memory limits
  4. Is it better to use the mapred or the mapreduce package to create a Hadoop Job?
  5. Chaining multiple MapReduce jobs in Hadoop
  6. When do reduce tasks start in Hadoop?
  7. How to get the input file name in the mapper in a Hadoop program?
  8. Hadoop speculative task execution
  9. Hive unable to manually set number of reducers
  10. how many mappers and reduces will get created for a partitoned table in hive
  11. hadoop map reduce secondary sorting
  12. Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)
  13. How does Hadoop perform input splits?
  14. What is the use of grouping comparator in hadoop map reduce
  15. Hadoop input split size vs block size
  16. Default number of reducers
  17. What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
  18. What is Google’s Dremel? How is it different from Mapreduce?
  19. Hadoop WordCount example stuck at map 100% reduce 0%
  20. Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?
  21. hadoop: difference between 0 reducer and identity reducer?
  22. Information about big data and hadoop [closed]
  23. How to delete and update a record in Hive
  24. Hadoop DistributedCache is deprecated – what is the preferred API?
  25. Hadoop Word count: receive the total number of words that start with the letter “c”
  26. Pyspark: get list of files/directories on HDFS path
  27. How to load data to hive from HDFS without removing the source file?
  28. Loading Data from a .txt file to Table Stored as ORC in Hive
  29. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  30. Datanode process not running in Hadoop

Leave a Comment