Spark read file from S3 using sc.textFile (“s3n://…)

by

Confirmed that this is related to the Spark build against Hadoop 2.60. Just installed Spark 1.4.0 “Pre built for Hadoop 2.4 and later” (instead of Hadoop 2.6). And the code now works OK.

sc.textFile("s3n://bucketname/Filename") now raises another error:

java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively).

The code below uses the S3 URL format to show that Spark can read S3 file. Using dev machine (no Hadoop libs).

scala> val lyrics = sc.textFile("s3n://MyAccessKeyID:MySecretKey@zpub01/SafeAndSound_Lyrics.txt")
lyrics: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[3] at textFile at <console>:21

scala> lyrics.count
res1: Long = 9

Even Better: the code above with AWS credentials inline in the S3N URI will break if the AWS Secret Key has a forward “https://stackoverflow.com/”. Configuring AWS Credentials in SparkContext will fix it. Code works whether the S3 file is public or private.

sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", "BLABLA")
sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", "....") // can contain "https://stackoverflow.com/"
val myRDD = sc.textFile("s3n://myBucket/MyFilePattern")
myRDD.count

More Related Contents:

  1. Matrix Multiplication in Apache Spark [closed]
  2. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  3. Resolving dependency problems in Apache Spark
  4. Add JAR files to a Spark job – spark-submit
  5. Spark: How to map Python with Scala or Java User Defined Functions?
  6. Spark Strutured Streaming automatically converts timestamp to local time
  7. How can I update a broadcast variable in spark streaming?
  8. Apache Spark – foreach Vs foreachPartition When to use What?
  9. What are the Spark transformations that causes a Shuffle?
  10. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  11. Is gzip format supported in Spark?
  12. Change output filename prefix for DataFrame.write()
  13. How can I force Spark to execute code?
  14. How to get element by Index in Spark RDD (Java)
  15. How to save models from ML Pipeline to S3 or HDFS?
  16. How to use java.String.format in Scala?
  17. Case class equality in Apache Spark
  18. Scala equivalent of Java java.lang.Class Object
  19. Is asynchronous jdbc call possible?
  20. Memory barriers and coding style over a Java VM
  21. Converting a Java collection into a Scala collection
  22. Java 8 Stream, getting head and tail
  23. How do I disambiguate in Scala between methods with vararg and without
  24. How can I use a Scala singleton object in Java?
  25. Using a Java library with Scala reserved words
  26. How do I call a UDF on a Spark DataFrame using JAVA?
  27. Which programming languages can be used to develop in Android? [duplicate]
  28. How to sort an RDD in Scala Spark?
  29. Why does SparkSession execute twice for one action?
  30. How to transpose an RDD in Spark

Leave a Comment