Behavior of the parameter “mapred.min.split.size” in HDFS

by

The split size is calculated by the formula:-

max(mapred.min.split.size, min(mapred.max.split.size, dfs.block.size))

In your case it will be:-

split size=max(128,min(Long.MAX_VALUE(default),64))

So above inference:-

  1. each map will process 2 hdfs blocks(assuming each block 64MB): True

  2. There will be a new division of my input file (previously included HDFS) to occupy blocks in HDFS 128M: False

but making the minimum split size greater than the block size increases the split size, but at the cost of locality.

More Related Contents:

  1. How does Hadoop process records split across block boundaries?
  2. How does Hadoop Namenode failover process works?
  3. Name node is in safe mode. Not able to leave
  4. How to fix corrupt HDFS FIles
  5. Hadoop java.io.IOException: Mkdirs failed to create /some/path
  6. Spark on yarn concept understanding
  7. Namenode not getting started
  8. Hive: Add partitions for existing folder structure
  9. Spark iterate HDFS directory
  10. Default Namenode port of HDFS is 50070.But I have come across at some places 8020 or 9000 [closed]
  11. How does Hadoop perform input splits?
  12. Hadoop: …be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation
  13. Default number of reducers
  14. Difference between HBase and Hadoop/HDFS
  15. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  16. Begenner at spark Big data programming (spark code)
  17. How to open/stream .zip files through Spark?
  18. Sqoop import : composite primary key and textual primary key
  19. When do reduce tasks start in Hadoop?
  20. data block size in HDFS, why 64MB?
  21. How can I access S3/S3n from a local Hadoop 2.6 installation?
  22. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  23. OOZIE: properties defined in file referenced in global job-xml not visible in workflow.xml
  24. Is it better to have one large parquet file or lots of smaller parquet files?
  25. hadoop map reduce secondary sorting
  26. Where are logs in Spark on YARN?
  27. Parallel Algorithms for Generating Prime Numbers (possibly using Hadoop’s map reduce)
  28. Hadoop input split size vs block size
  29. Job queue for Hive action in oozie
  30. What is Hive: Return Code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask

Leave a Comment