Reading DataFrame from partitioned parquet file

by

sqlContext.read.parquet can take multiple paths as input. If you want just day=5 and day=6, you can simply add two paths like:

val dataframe = sqlContext
      .read.parquet("file:///your/path/data=jDD/year=2015/month=10/day=5/", 
                    "file:///your/path/data=jDD/year=2015/month=10/day=6/")

If you have folders under day=X, like say country=XX, country will automatically be added as a column in the dataframe.

EDIT: As of Spark 1.6 one needs to provide a “basepath”-option in order for Spark to generate columns automatically. In Spark 1.6.x the above would have to be re-written like this to create a dataframe with the columns “data”, “year”, “month” and “day”:

val dataframe = sqlContext
     .read
     .option("basePath", "file:///your/path/")
     .parquet("file:///your/path/data=jDD/year=2015/month=10/day=5/", 
                    "file:///your/path/data=jDD/year=2015/month=10/day=6/")

More Related Contents:

  1. Spark DataFrame: does groupBy after orderBy maintain that order?
  2. Change nullable property of column in spark dataframe
  3. (Why) do we need to call cache or persist on a RDD
  4. How to load local file in sc.textFile, instead of HDFS
  5. Automatically and Elegantly flatten DataFrame in Spark SQL
  6. How do I skip a header from CSV files in Spark?
  7. Spark extracting values from a Row
  8. How to pass -D parameter or environment variable to Spark job?
  9. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  10. Append a column to Data Frame in Apache Spark 1.3
  11. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  12. Processing multiple files as independent RDD’s in parallel
  13. MatchError while accessing vector column in Spark 2.0
  14. Spark: what’s the best strategy for joining a 2-tuple-key RDD with single-key RDD?
  15. Spark / Scala: forward fill with last observation
  16. Function returns an empty List in Spark
  17. How to compare two dataframe and print columns that are different in scala
  18. Operate on neighbor elements in RDD in Spark
  19. Renaming column names of a DataFrame in Spark Scala
  20. What is going wrong with `unionAll` of Spark `DataFrame`?
  21. Spark: long delay between jobs
  22. How to get keys and values from MapType column in SparkSQL DataFrame
  23. How to compute cumulative sum using Spark
  24. Addition of two RDD[mllib.linalg.Vector]’s
  25. Add Number of days column to Date Column in same dataframe for Spark Scala App
  26. How to sort an RDD in Scala Spark?
  27. Difference between two rows in Spark dataframe
  28. Run a scala code jar appear NoSuchMethodError:scala.Predef$.refArrayOps
  29. Spark UDF with varargs
  30. How to transpose an RDD in Spark

Leave a Comment