Chaining multiple MapReduce jobs in Hadoop

by

I think this tutorial on Yahoo’s developer network will help you with this: Chaining Jobs

You use the JobClient.runJob(). The output path of the data from the first job becomes the input path to your second job. These need to be passed in as arguments to your jobs with appropriate code to parse them and set up the parameters for the job.

I think that the above method might however be the way the now older mapred API did it, but it should still work. There will be a similar method in the new mapreduce API but i’m not sure what it is.

As far as removing intermediate data after a job has finished you can do this in your code. The way i’ve done it before is using something like:

FileSystem.delete(Path f, boolean recursive);

Where the path is the location on HDFS of the data. You need to make sure that you only delete this data once no other job requires it.

More Related Contents:

  1. How does Hadoop process records split across block boundaries?
  2. merge output files after reduce phase
  3. Container is running beyond memory limits
  4. Setting the number of map tasks and reduce tasks
  5. Is it better to use the mapred or the mapreduce package to create a Hadoop Job?
  6. When do reduce tasks start in Hadoop?
  7. How to get the input file name in the mapper in a Hadoop program?
  8. Hadoop speculative task execution
  9. Hive unable to manually set number of reducers
  10. how many mappers and reduces will get created for a partitoned table in hive
  11. hadoop map reduce secondary sorting
  12. Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)
  13. How does Hadoop perform input splits?
  14. What is the use of grouping comparator in hadoop map reduce
  15. Hadoop input split size vs block size
  16. Default number of reducers
  17. What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
  18. What is Google’s Dremel? How is it different from Mapreduce?
  19. Hadoop WordCount example stuck at map 100% reduce 0%
  20. Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?
  21. hadoop: difference between 0 reducer and identity reducer?
  22. Information about big data and hadoop [closed]
  23. Calling a mapreduce job from a simple java program
  24. What is the purpose of shuffling and sorting phase in the reducer in Map Reduce Programming?
  25. How to delete and update a record in Hive
  26. Hadoop DistributedCache is deprecated – what is the preferred API?
  27. Hadoop Word count: receive the total number of words that start with the letter “c”
  28. Pyspark: get list of files/directories on HDFS path
  29. How to load data to hive from HDFS without removing the source file?
  30. Loading Data from a .txt file to Table Stored as ORC in Hive

Leave a Comment