data block size in HDFS, why 64MB?

by

What does 64MB block size mean?

The block size is the smallest data unit that a file system can store. If you store a file that’s 1k or 60Mb, it’ll take up one block. Once you cross the 64Mb boundary, you need a second block.

If yes, what is the advantage of doing that?

HDFS is meant to handle large files. Let’s say you have a 1000Mb file. With a 4k block size, you’d have to make 256,000 requests to get that file (1 request per block). In HDFS, those requests go across a network and come with a lot of overhead. Each request has to be processed by the Name Node to determine where that block can be found. That’s a lot of traffic! If you use 64Mb blocks, the number of requests goes down to 16, significantly reducing the cost of overhead and load on the Name Node.

More Related Contents:

  1. How does Hadoop process records split across block boundaries?
  2. Primary keys with Apache Spark
  3. What is the purpose of shuffling and sorting phase in the reducer in Map Reduce Programming?
  4. How do I output the results of a HiveQL query to CSV?
  5. How does Hadoop perform input splits?
  6. Default number of reducers
  7. Normalization in database management system
  8. How to perform compound queries with logical OR in Cloud Firestore?
  9. Database development mistakes made by application developers [closed]
  10. Copying PostgreSQL database to another server
  11. What is NoSQL, how does it work, and what benefits does it provide? [closed]
  12. First name, middle name, last name. Why not Full Name?
  13. Can we update primary key values of a table?
  14. What are the design criteria for primary keys?
  15. Oracle (ORA-02270) : no matching unique or primary key for this column-list error
  16. SQLite UPSERT / UPDATE OR INSERT
  17. hadoop.mapred vs hadoop.mapreduce?
  18. Elastic search, multiple indexes vs one index and types for different data sets?
  19. Spark on yarn concept understanding
  20. Good Free Alternative To MS Access [closed]
  21. How to get the input file name in the mapper in a Hadoop program?
  22. What’s wrong with nullable columns in composite primary keys?
  23. Permission denied at hdfs
  24. Convert String ISO-8601 date to oracle’s timestamp datatype
  25. Are Cloud Firestore queries still case sensitive?
  26. How should you build your database from source control?
  27. Where does hadoop mapreduce framework send my System.out.print() statements ? (stdout)
  28. Strange behavior in PowerShell function returning DataSet/DataTable
  29. CAP theorem – Availability and Partition Tolerance
  30. Store images/videos into Hadoop HDFS

Leave a Comment