Is gzip format supported in Spark?

by

From the Spark Scala Programming guide’s section on “Hadoop Datasets”:

Spark can create distributed datasets from any file stored in the Hadoop distributed file system (HDFS) or other storage systems supported by Hadoop (including your local file system, Amazon S3, Hypertable, HBase, etc). Spark supports text files, SequenceFiles, and any other Hadoop InputFormat.

Support for gzip input files should work the same as it does in Hadoop. For example, sc.textFile("myFile.gz") should automatically decompress and read gzip-compressed files (textFile() is actually implemented using Hadoop’s TextInputFormat, which supports gzip-compressed files).

As mentioned by @nick-chammas in the comments:

note that if you call sc.textFile() on a gzipped file, Spark will give
you an RDD with only 1 partition (as of 0.9.0). This is because
gzipped files are not splittable. If you don’t repartition the RDD
somehow, any operations on that RDD will be limited to a single core

More Related Contents:

  1. Change output filename prefix for DataFrame.write()
  2. Resolving dependency problems in Apache Spark
  3. Add JAR files to a Spark job – spark-submit
  4. Spark: How to map Python with Scala or Java User Defined Functions?
  5. Spark read file from S3 using sc.textFile (“s3n://…)
  6. Spark Strutured Streaming automatically converts timestamp to local time
  7. How can I update a broadcast variable in spark streaming?
  8. Apache Spark – foreach Vs foreachPartition When to use What?
  9. Matrix Multiplication in Apache Spark [closed]
  10. What are the Spark transformations that causes a Shuffle?
  11. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  12. How can I force Spark to execute code?
  13. How to save models from ML Pipeline to S3 or HDFS?
  14. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  15. How to do aggregate functions, may columns and extract back
  16. Jvm takes a long time to resolve ip-address for localhost
  17. Using GZIP compression with Spring Boot/MVC/JavaConfig with RESTful
  18. “eval” in Scala
  19. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  20. Get the changed HTML content after it’s updated by Javascript? (htmlunit)
  21. Is there a PriorityQueue implementation with fixed capacity and custom comparator?
  22. Timestamp convert [duplicate]
  23. Writing Custom Kafka Serializer
  24. Iterating over Java collections in Scala
  25. PySpark: java.lang.OutofMemoryError: Java heap space
  26. java.lang.NoClassDefFoundError: org/apache/spark/Logging
  27. Convert Java Map to Scala Map
  28. Creating a jar file from a Scala file
  29. Spark: get number of cluster cores programmatically
  30. Why does SparkSession execute twice for one action?

Leave a Comment