Reading parquet files from multiple directories in Pyspark

by

A little late but I found this while I was searching and it may help someone else…

You might also try unpacking the argument list to spark.read.parquet()

paths=['foo','bar']
df=spark.read.parquet(*paths)

This is convenient if you want to pass a few blobs into the path argument:

basePath="s3://bucket/"
paths=['s3://bucket/partition_value1=*/partition_value2=2017-04-*',
       's3://bucket/partition_value1=*/partition_value2=2017-05-*'
      ]
df=spark.read.option("basePath",basePath).parquet(*paths)

This is cool cause you don’t need to list all the files in the basePath, and you still get partition inference.

More Related Contents:

  1. How to make good reproducible Apache Spark examples
  2. Spark Dataframe validating column names for parquet writes
  3. Applying a Window function to calculate differences in pySpark
  4. Manually create a pyspark dataframe
  5. combine text from multiple rows in pyspark
  6. Using a column value as a parameter to a spark DataFrame function
  7. How to split Vector into columns – using PySpark
  8. java.lang.IllegalArgumentException at org.apache.xbean.asm5.ClassReader.(Unknown Source) with Java 10
  9. Split Spark Dataframe string column into multiple columns
  10. Spark: How to map Python with Scala or Java User Defined Functions?
  11. creating spark data structure from multiline record
  12. Pyspark: explode json in column to multiple columns
  13. How to split a list to multiple columns in Pyspark?
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How do you control the size of the output file?
  16. How to explode multiple columns of a dataframe in pyspark
  17. Efficient pyspark join
  18. PySpark: multiple conditions in when clause
  19. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  20. PySpark – get row number for each row in a group
  21. Save ML model for future usage
  22. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  23. Keep only duplicates from a DataFrame regarding some field
  24. What are the differences between feather and parquet?
  25. ImportError: No module named numpy on spark workers
  26. How do I convert an array (i.e. list) column to Vector
  27. pyspark: rolling average using timeseries data
  28. Add column sum as new column in PySpark dataframe
  29. How to drop all columns with null values in a PySpark DataFrame?
  30. Spark ALS predictAll returns empty

Leave a Comment