java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics

by

You can’t mix bits of Hadoop and expect things to work. It’s not just the close coupling between internal classes in hadoop-common and hadoop-aws, its things like the specific version of the amazon-aws SDK the hadoop-aws module was built it.

If you get ClassNotFoundException or MethodNotFoundException stack traces when trying to work with s3a:// URLs, JAR version mismatch is the likely cause.

Using the RFC2117 MUST/SHOULD/MAY terminology, here are the rules to avoid this situation:

  1. The s3a connector is in hadoop-aws JAR; it depends on hadoop-common and the aws-sdk-shaded JARs.
  2. all these JARs MUST be on the classpath.
  3. All versions of the hadoop-* JARs on your classpath MUST be exactly the same version, e.g 3.3.1 everywhere, or 3.2.2. Otherwise: stack trace. Always
  4. And they MUST be exclusively of that version; there MUST NOT be multiple versions of hadoop-common, hadoop-aws etc on the classpath. Otherwise: stack trace. Always. Usually ClassNotFoundException indicating a mismatch in hadoop-common and hadoop-aws.
  5. The exact missing class varies across Hadoop releases: it’s the first class depended on by org.apache.fs.s3a.S3AFileSystem which the classloader can’t find -the exact class depends on the mismatch of JARs
  6. The AWS SDK version SHOULD be the one shipped. Otherwise: maybe stack trace, maybe not. Either way -you are in self-support mode or have opted to join a QE team for version testing.
  7. The specific version of the AWS SDK you need can be determined from Maven Repository
  8. Changing the AWS SDK versions MAY work. You get to test, and if there are compatibility problems: you get to fix. See Qualifying an AWS SDK Update for the least you should be doing.
  9. You SHOULD use the most recent versions of Hadoop you can/Spark is tested with. Non-critical bug fixes do not get backported to old Hadoop releases, and the S3A and ABFS connectors are rapidly evolving. New releases will be better, stronger, faster. Generally
  10. If none of this works. a bug report filed on the ASF JIRA server will get closed as WORKSFORME. Config issues aren’t treated as code bugs

Finally: the ASF documentationThe S3A Connector

More Related Contents:

  1. Begenner at spark Big data programming (spark code)
  2. How to open/stream .zip files through Spark?
  3. Spark on yarn concept understanding
  4. Pyspark: get list of files/directories on HDFS path
  5. How to access s3a:// files from Apache Spark?
  6. Is it better to have one large parquet file or lots of smaller parquet files?
  7. Apache Spark: The number of cores vs. the number of executors
  8. Apache Hadoop Yarn – Underutilization of cores
  9. Where are logs in Spark on YARN?
  10. Easiest way to install Python dependencies on Spark executor nodes?
  11. Spark iterate HDFS directory
  12. Cannot Read a file from HDFS using Spark
  13. Primary keys with Apache Spark
  14. Failed to locate the winutils binary in the hadoop binary path
  15. What is the difference between partitioning and bucketing a table in Hive ?
  16. Requirements for converting Spark dataframe to Pandas/R dataframe
  17. How can PySpark be called in debug mode?
  18. Sqoop import : composite primary key and textual primary key
  19. When do reduce tasks start in Hadoop?
  20. Hadoop java.io.IOException: Mkdirs failed to create /some/path
  21. How can I access S3/S3n from a local Hadoop 2.6 installation?
  22. OOZIE: properties defined in file referenced in global job-xml not visible in workflow.xml
  23. How can I force Spark to execute code?
  24. Parallel Algorithms for Generating Prime Numbers (possibly using Hadoop’s map reduce)
  25. Spark Scala list folders in directory
  26. Difference between Pig and Hive? Why have both? [closed]
  27. Difference between hadoop fs -put and hadoop fs -copyFromLocal
  28. Datanode process not running in Hadoop
  29. Behavior of the parameter “mapred.min.split.size” in HDFS
  30. Hive Data Retrieval Queries: Difference between CLUSTER BY, ORDER BY, and SORT BY

Leave a Comment