How can I access S3/S3n from a local Hadoop 2.6 installation?

by

For some reason, the jar hadoop-aws-[version].jar which contains the implementation to NativeS3FileSystem is not present in the classpath of hadoop by default in the version 2.6 & 2.7. So, try and add it to the classpath by adding the following line in hadoop-env.sh which is located in $HADOOP_HOME/etc/hadoop/hadoop-env.sh:

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_HOME/share/hadoop/tools/lib/*

Assuming you are using Apache Hadoop 2.6 or 2.7

By the way, you could check the classpath of Hadoop using:

bin/hadoop classpath

More Related Contents:

  1. Container is running beyond memory limits
  2. How does Hadoop Namenode failover process works?
  3. Spark on yarn concept understanding
  4. How to access s3a:// files from Apache Spark?
  5. Apache Spark: The number of cores vs. the number of executors
  6. Apache Hadoop Yarn – Underutilization of cores
  7. Where are logs in Spark on YARN?
  8. Begenner at spark Big data programming (spark code)
  9. Failed to locate the winutils binary in the hadoop binary path
  10. What is the difference between partitioning and bucketing a table in Hive ?
  11. react router doesn’t work in aws s3 bucket
  12. How to write a file or data to an S3 object using boto3
  13. How to open/stream .zip files through Spark?
  14. Setting up FTP on Amazon Cloud Server [closed]
  15. How to upload a file to directory in S3 bucket using boto
  16. Nginx proxy Amazon S3 resources
  17. Serving a multitude of static sites from a wildcard domain in AWS
  18. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  19. Is it better to have one large parquet file or lots of smaller parquet files?
  20. hadoop map reduce secondary sorting
  21. How to list all AWS S3 objects in a bucket using Java
  22. Getting S3 objects’ last modified datetimes with boto
  23. Amazon S3 Redirect and Cloudfront
  24. Parallel Algorithms for Generating Prime Numbers (possibly using Hadoop’s map reduce)
  25. How do you search an amazon s3 bucket?
  26. Hadoop input split size vs block size
  27. AWS Missing credentials when I try send something to my S3 Bucket (Node.js)
  28. AWS Glue output file name
  29. AWS CLI S3 A client error (403) occurred when calling the HeadObject operation: Forbidden
  30. Download a folder from S3 using Boto3

Leave a Comment