How to access s3a:// files from Apache Spark?

by

Having experienced first hand the difference between s3a and s3n – 7.9GB of data transferred on s3a was around ~7 minutes while 7.9GB of data on s3n took 73 minutes [us-east-1 to us-west-1 unfortunately in both cases; Redshift and Lambda being us-east-1 at this time] this is a very important piece of the stack to get correct and it’s worth the frustration.

Here are the key parts, as of December 2015:

  1. Your Spark cluster will need a Hadoop version 2.x or greater. If you use the Spark EC2 setup scripts and maybe missed it, the switch for using something other than 1.0 is to specify --hadoop-major-version 2 (which uses CDH 4.2 as of this writing).

  2. You’ll need to include what may at first seem to be an out of date AWS SDK library (built in 2014 as version 1.7.4) for versions of Hadoop as late as 2.7.1 (stable): aws-java-sdk 1.7.4. As far as I can tell using this along with the specific AWS SDK JARs for 1.10.8 hasn’t broken anything.

  3. You’ll also need the hadoop-aws 2.7.1 JAR on the classpath. This JAR contains the class org.apache.hadoop.fs.s3a.S3AFileSystem.

  4. In spark.properties you probably want some settings that look like this:

    spark.hadoop.fs.s3a.access.key=ACCESSKEY spark.hadoop.fs.s3a.secret.key=SECRETKEY

  5. If you are using hadoop 2.7 version with spark then the aws client uses V2 as default auth signature. And all the new aws region support only V4 protocol. To use V4 pass these conf in spark-submit and also endpoint (format – s3.<region>.amazonaws.com) must be specified.

--conf "spark.executor.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true

--conf "spark.driver.extraJavaOptions=-Dcom.amazonaws.services.s3.enableV4=true

I’ve detailed this list in more detail on a post I wrote as I worked my way through this process. In addition I’ve covered all the exception cases I hit along the way and what I believe to be the cause of each and how to fix them.

More Related Contents:

  1. Begenner at spark Big data programming (spark code)
  2. How to open/stream .zip files through Spark?
  3. Spark on yarn concept understanding
  4. How can I access S3/S3n from a local Hadoop 2.6 installation?
  5. Pyspark: get list of files/directories on HDFS path
  6. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  7. Is it better to have one large parquet file or lots of smaller parquet files?
  8. Apache Spark: The number of cores vs. the number of executors
  9. Apache Hadoop Yarn – Underutilization of cores
  10. Where are logs in Spark on YARN?
  11. Easiest way to install Python dependencies on Spark executor nodes?
  12. Spark iterate HDFS directory
  13. Cannot Read a file from HDFS using Spark
  14. Information about big data and hadoop [closed]
  15. Write to multiple outputs by key Spark – one Spark job
  16. Spark – load CSV file as DataFrame?
  17. Read whole text files from a compression in Spark
  18. Spark: Inconsistent performance number in scaling number of cores
  19. Container is running beyond memory limits
  20. Chaining multiple MapReduce jobs in Hadoop
  21. Apache Spark: Get number of records per partition
  22. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  23. how many mappers and reduces will get created for a partitoned table in hive
  24. How to load data to hive from HDFS without removing the source file?
  25. Loading Data from a .txt file to Table Stored as ORC in Hive
  26. Pig Latin: Load multiple files from a date range (part of the directory structure)
  27. Spark lists all leaf node even in partitioned data
  28. What is the use of grouping comparator in hadoop map reduce
  29. How to read input from S3 in a Spark Streaming EC2 cluster application
  30. hadoop: difference between 0 reducer and identity reducer?

Leave a Comment