How does Hadoop perform input splits?

by

The InputFormat is responsible to provide the splits.

In general, if you have n nodes, the HDFS will distribute the file over all these n nodes. If you start a job, there will be n mappers by default. Thanks to Hadoop, the mapper on a machine will process the part of the data that is stored on this node. I think this is called Rack awareness.

So to make a long story short: Upload the data in the HDFS and start a MR Job. Hadoop will care for the optimised execution.

More Related Contents:

  1. How does Hadoop process records split across block boundaries?
  2. Default number of reducers
  3. Container is running beyond memory limits
  4. How does Hadoop Namenode failover process works?
  5. What is the purpose of shuffling and sorting phase in the reducer in Map Reduce Programming?
  6. Name node is in safe mode. Not able to leave
  7. How to fix corrupt HDFS FIles
  8. Chaining multiple MapReduce jobs in Hadoop
  9. When do reduce tasks start in Hadoop?
  10. Hadoop java.io.IOException: Mkdirs failed to create /some/path
  11. Spark on yarn concept understanding
  12. data block size in HDFS, why 64MB?
  13. How to get the input file name in the mapper in a Hadoop program?
  14. Hadoop speculative task execution
  15. how many mappers and reduces will get created for a partitoned table in hive
  16. hadoop map reduce secondary sorting
  17. Namenode not getting started
  18. Hive: Add partitions for existing folder structure
  19. Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)
  20. Default Namenode port of HDFS is 50070.But I have come across at some places 8020 or 9000 [closed]
  21. Hadoop: …be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation
  22. What is the use of grouping comparator in hadoop map reduce
  23. Hadoop input split size vs block size
  24. Difference between HBase and Hadoop/HDFS
  25. What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
  26. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  27. What is Google’s Dremel? How is it different from Mapreduce?
  28. Hadoop WordCount example stuck at map 100% reduce 0%
  29. Behavior of the parameter “mapred.min.split.size” in HDFS
  30. hadoop: difference between 0 reducer and identity reducer?

Leave a Comment