How to open/stream .zip files through Spark?

by

There was no solution with python code and I recently had to read zips in pyspark. And, while searching how to do that I came across this question. So, hopefully this’ll help others.

import zipfile
import io

def zip_extract(x):
    in_memory_data = io.BytesIO(x[1])
    file_obj = zipfile.ZipFile(in_memory_data, "r")
    files = [i for i in file_obj.namelist()]
    return dict(zip(files, [file_obj.open(file).read() for file in files]))


zips = sc.binaryFiles("hdfs:/Testing/*.zip")
files_data = zips.map(zip_extract).collect()

In the above code I returned a dictionary with filename in the zip as a key and the text data in each file as the value. you can change it however you want to suit your purposes.

More Related Contents:

  1. Begenner at spark Big data programming (spark code)
  2. Spark on yarn concept understanding
  3. Pyspark: get list of files/directories on HDFS path
  4. java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
  5. How to access s3a:// files from Apache Spark?
  6. Is it better to have one large parquet file or lots of smaller parquet files?
  7. Apache Spark: The number of cores vs. the number of executors
  8. Apache Hadoop Yarn – Underutilization of cores
  9. Where are logs in Spark on YARN?
  10. Easiest way to install Python dependencies on Spark executor nodes?
  11. Spark iterate HDFS directory
  12. Cannot Read a file from HDFS using Spark
  13. Information about big data and hadoop [closed]
  14. Write to multiple outputs by key Spark – one Spark job
  15. Spark – load CSV file as DataFrame?
  16. Read whole text files from a compression in Spark
  17. Spark: Inconsistent performance number in scaling number of cores
  18. Container is running beyond memory limits
  19. Name node is in safe mode. Not able to leave
  20. How to fix corrupt HDFS FIles
  21. How to delete and update a record in Hive
  22. What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism?
  23. Apache Spark: Get number of records per partition
  24. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  25. how many mappers and reduces will get created for a partitoned table in hive
  26. How to load data to hive from HDFS without removing the source file?
  27. Loading Data from a .txt file to Table Stored as ORC in Hive
  28. What is the use of grouping comparator in hadoop map reduce
  29. Hadoop WordCount example stuck at map 100% reduce 0%
  30. Can Hive recursively descend into subdirectories without partitions or editing hive-site.xml?

Leave a Comment