Hadoop: …be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and no node(s) are excluded in this operation

by

This error is caused by the block replication system of HDFS since it could not manage to make any copies of a specific block within the focused file. Common reasons of that:

  1. Only a NameNode instance is running and it’s not in safe-mode
  2. There is no DataNode instances up and running, or some are dead. (Check the servers)
  3. Namenode and Datanode instances are both running, but they cannot communicate with each other, which means There is connectivity issue between DataNode and NameNode instances.
  4. Running DataNode instances are not able to talk to the server because of some networking of hadoop-based issues (check logs that include datanode info)
  5. There is no hard disk space specified in configured data directories for DataNode instances or DataNode instances have run out of space. (check dfs.data.dir // delete old files if any)
  6. Specified reserved spaces for DataNode instances in dfs.datanode.du.reserved is more than the free space which makes DataNode instances to understand there is no enough free space.
  7. There is no enough threads for DataNode instances (check datanode logs and dfs.datanode.handler.count value)
  8. Make sure dfs.data.transfer.protection is not equal to “authentication” and dfs.encrypt.data.transfer is equal to true.

Also please:

  • Verify the status of NameNode and DataNode services and check the related logs
  • Verify if core-site.xml has correct fs.defaultFS value and hdfs-site.xml has a valid value.
  • Verify hdfs-site.xml has dfs.namenode.http-address.. for all NameNode instances specified in case of PHD HA configuration.
  • Verify if the permissions on the directories are correct

Ref: https://wiki.apache.org/hadoop/CouldOnlyBeReplicatedTo

Ref: https://support.pivotal.io/hc/en-us/articles/201846688-HDFS-reports-Configured-Capacity-0-0-B-for-datanode

Also, please check: Writing to HDFS from Java, getting “could only be replicated to 0 nodes instead of minReplication”

More Related Contents:

  1. How does Hadoop process records split across block boundaries?
  2. How does Hadoop Namenode failover process works?
  3. Name node is in safe mode. Not able to leave
  4. How to fix corrupt HDFS FIles
  5. Hadoop java.io.IOException: Mkdirs failed to create /some/path
  6. Spark on yarn concept understanding
  7. Namenode not getting started
  8. Hive: Add partitions for existing folder structure
  9. Spark iterate HDFS directory
  10. Default Namenode port of HDFS is 50070.But I have come across at some places 8020 or 9000 [closed]
  11. How does Hadoop perform input splits?
  12. Default number of reducers
  13. Difference between HBase and Hadoop/HDFS
  14. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  15. Datanode process not running in Hadoop
  16. Behavior of the parameter “mapred.min.split.size” in HDFS
  17. Information about big data and hadoop [closed]
  18. Write to multiple outputs by key Spark – one Spark job
  19. Spark – load CSV file as DataFrame?
  20. Read whole text files from a compression in Spark
  21. Container is running beyond memory limits
  22. What is the purpose of shuffling and sorting phase in the reducer in Map Reduce Programming?
  23. Chaining multiple MapReduce jobs in Hadoop
  24. Pyspark: get list of files/directories on HDFS path
  25. how many mappers and reduces will get created for a partitoned table in hive
  26. How to load data to hive from HDFS without removing the source file?
  27. What is the use of grouping comparator in hadoop map reduce
  28. Difference between Pig and Hive? Why have both? [closed]
  29. What is Google’s Dremel? How is it different from Mapreduce?
  30. Hive Data Retrieval Queries: Difference between CLUSTER BY, ORDER BY, and SORT BY

Leave a Comment