Hadoop input split size vs block size

by

  • The answer by @user1668782 is a great explanation for the question and I’ll try to give a graphical depiction of it.

  • Assume we have a file of 400MB with consists of 4 records(e.g : csv file of 400MB and it has 4 rows, 100MB each)

enter image description here

  • If the HDFS Block Size is configured as 128MB, then the 4 records will not be distributed among the blocks evenly. It will look like this.

enter image description here

  • Block 1 contains the entire first record and a 28MB chunk of the second record.
  • If a mapper is to be run on Block 1, the mapper cannot process since it won’t have the entire second record.

  • This is the exact problem that input splits solve. Input splits respects logical record boundaries.

  • Lets Assume the input split size is 200MB

enter image description here

  • Therefore the input split 1 should have both the record 1 and record 2. And input split 2 will not start with the record 2 since record 2 has been assigned to input split 1. Input split 2 will start with record 3.

  • This is why an input split is only a logical chunk of data. It points to start and end locations with in blocks.

Hope this helps.

More Related Contents:

  1. How does Hadoop process records split across block boundaries?
  2. merge output files after reduce phase
  3. Container is running beyond memory limits
  4. Setting the number of map tasks and reduce tasks
  5. Is it better to use the mapred or the mapreduce package to create a Hadoop Job?
  6. Chaining multiple MapReduce jobs in Hadoop
  7. When do reduce tasks start in Hadoop?
  8. How to get the input file name in the mapper in a Hadoop program?
  9. Hadoop speculative task execution
  10. Hive unable to manually set number of reducers
  11. how many mappers and reduces will get created for a partitoned table in hive
  12. hadoop map reduce secondary sorting
  13. Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)
  14. How does Hadoop perform input splits?
  15. What is the use of grouping comparator in hadoop map reduce
  16. Default number of reducers
  17. What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
  18. What is Google’s Dremel? How is it different from Mapreduce?
  19. Hadoop WordCount example stuck at map 100% reduce 0%
  20. Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?
  21. hadoop: difference between 0 reducer and identity reducer?
  22. Begenner at spark Big data programming (spark code)
  23. Failed to locate the winutils binary in the hadoop binary path
  24. What is the difference between partitioning and bucketing a table in Hive ?
  25. Oozie: Launch Map-Reduce from Oozie action?
  26. data block size in HDFS, why 64MB?
  27. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  28. Is it better to have one large parquet file or lots of smaller parquet files?
  29. Easiest way to install Python dependencies on Spark executor nodes?
  30. Change File Split size in Hadoop

Leave a Comment