Hive unable to manually set number of reducers

by

writing query in hive like this:

 SELECT COUNT(DISTINCT id) ....

will always result in using only one reducer.
You should:

  1. use this command to set desired number of reducers:

    set mapred.reduce.tasks=50

  2. rewrite query as following:

SELECT COUNT(*) FROM ( SELECT DISTINCT id FROM … ) t;

This will result in 2 map+reduce jobs instead of one, but performance gain will be substantial.

More Related Contents:

  1. how many mappers and reduces will get created for a partitoned table in hive
  2. What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
  3. Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?
  4. How does Hadoop process records split across block boundaries?
  5. merge output files after reduce phase
  6. What is the difference between partitioning and bucketing a table in Hive ?
  7. Setting the number of map tasks and reduce tasks
  8. Difference between Hive internal tables and external tables?
  9. How to transpose/pivot data in hive?
  10. Is it better to use the mapred or the mapreduce package to create a Hadoop Job?
  11. Sqoop import : composite primary key and textual primary key
  12. Chaining multiple MapReduce jobs in Hadoop
  13. When do reduce tasks start in Hadoop?
  14. How to delete and update a record in Hive
  15. How to get the input file name in the mapper in a Hadoop program?
  16. Hadoop speculative task execution
  17. How to update table in Hive 0.13?
  18. hadoop map reduce secondary sorting
  19. Hive: Add partitions for existing folder structure
  20. Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)
  21. How to load data to hive from HDFS without removing the source file?
  22. How does Hadoop perform input splits?
  23. Hadoop input split size vs block size
  24. Default number of reducers
  25. Create HIVE Table with multi character delimiter
  26. Job queue for Hive action in oozie
  27. Difference between Pig and Hive? Why have both? [closed]
  28. What is Google’s Dremel? How is it different from Mapreduce?
  29. Hadoop WordCount example stuck at map 100% reduce 0%
  30. Hive Data Retrieval Queries: Difference between CLUSTER BY, ORDER BY, and SORT BY

Leave a Comment