how many mappers and reduces will get created for a partitoned table in hive

by

Mappers:

Number of mappers depends on various factors such as how the data is distributed among nodes, input format, execution engine and configuration params. See also here: https://cwiki.apache.org/confluence/display/TEZ/How+initial+task+parallelism+works

MR uses CombineInputFormat, while Tez uses grouped splits.

Tez:

set tez.grouping.min-size=16777216; -- 16 MB min split
set tez.grouping.max-size=1073741824; -- 1 GB max split

MapReduce:

set mapreduce.input.fileinputformat.split.minsize=16777216; -- 16 MB
set mapreduce.input.fileinputformat.split.maxsize=1073741824; -- 1 GB

Also Mappers are running on data nodes where the data is located, that is why manually controlling the number of mappers is not an easy task, not always possible to combine input.

Reducers:
Controlling the number of reducers is much easier.
The number of reducers determined according to

mapreduce.job.reduces – The default number of reduce tasks per job. Typically set to a prime close to the number of available hosts. Ignored when mapred.job.tracker is “local”. Hadoop set this to 1 by default, whereas Hive uses -1 as its default value. By setting this property to -1, Hive will automatically figure out what should be the number of reducers.

hive.exec.reducers.bytes.per.reducer – The default in Hive 0.14.0 and earlier is 1 GB.

Also hive.exec.reducers.max – Maximum number of reducers that will be used. If mapreduce.job.reduces is negative, Hive will use this as the maximum number of reducers when automatically determining the number of reducers.

So, if you want to increase reducers parallelism, increase hive.exec.reducers.max and decrease hive.exec.reducers.bytes.per.reducer

More Related Contents:

  1. When do reduce tasks start in Hadoop?
  2. Hive unable to manually set number of reducers
  3. What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
  4. Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?
  5. How does Hadoop process records split across block boundaries?
  6. merge output files after reduce phase
  7. What is the difference between partitioning and bucketing a table in Hive ?
  8. Setting the number of map tasks and reduce tasks
  9. Difference between Hive internal tables and external tables?
  10. How to transpose/pivot data in hive?
  11. Is it better to use the mapred or the mapreduce package to create a Hadoop Job?
  12. Sqoop import : composite primary key and textual primary key
  13. Chaining multiple MapReduce jobs in Hadoop
  14. How to delete and update a record in Hive
  15. How to get the input file name in the mapper in a Hadoop program?
  16. Hadoop speculative task execution
  17. How to update table in Hive 0.13?
  18. hadoop map reduce secondary sorting
  19. Hive: Add partitions for existing folder structure
  20. Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)
  21. How to load data to hive from HDFS without removing the source file?
  22. How does Hadoop perform input splits?
  23. Hadoop input split size vs block size
  24. Default number of reducers
  25. Create HIVE Table with multi character delimiter
  26. Job queue for Hive action in oozie
  27. Difference between Pig and Hive? Why have both? [closed]
  28. What is Google’s Dremel? How is it different from Mapreduce?
  29. Hadoop WordCount example stuck at map 100% reduce 0%
  30. Hive Data Retrieval Queries: Difference between CLUSTER BY, ORDER BY, and SORT BY

Leave a Comment