How to import multiple csv files in a single load?

by

Use wildcard, e.g. replace 2008 with *:

df = sqlContext.read
       .format("com.databricks.spark.csv")
       .option("header", "true")
       .load("../Downloads/*.csv") // <-- note the star (*)

Spark 2.0

// these lines are equivalent in Spark 2.0
spark.read.format("csv").option("header", "true").load("../Downloads/*.csv")
spark.read.option("header", "true").csv("../Downloads/*.csv")

Notes:

  1. Replace format("com.databricks.spark.csv") by using format("csv") or csv method instead. com.databricks.spark.csv format has been integrated to 2.0.

  2. Use spark not sqlContext

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Overwrite specific partitions in spark dataframe write method
  3. How to save/insert each DStream into a permanent table
  4. How to melt Spark DataFrame?
  5. Spark SQL: apply aggregate functions to a list of columns
  6. Unpivot in spark-sql/pyspark
  7. Spark SQL replacement for MySQL’s GROUP_CONCAT aggregate function
  8. DataFrame join optimization – Broadcast Hash Join
  9. Spark SQL – load data with JDBC using SQL statement, not table name
  10. Spark Dataframe validating column names for parquet writes
  11. How to check if spark dataframe is empty?
  12. How to improve performance for slow Spark jobs using DataFrame and JDBC connection?
  13. TypeError: Column is not iterable – How to iterate over ArrayType()?
  14. How to connect to remote hive server from spark [duplicate]
  15. Spark DataFrame: does groupBy after orderBy maintain that order?
  16. Filtering a spark dataframe based on date
  17. Pyspark : forward fill with last observation for a DataFrame
  18. Spark DataFrame Schema Nullable Fields
  19. multiple conditions for filter in spark data frames
  20. How do I add an persistent column of row ids to Spark DataFrame?
  21. Rename more than one column using withColumnRenamed
  22. Convert null values to empty array in Spark DataFrame
  23. DataFrame partitionBy to a single Parquet file (per partition)
  24. What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?
  25. Temp table caching with spark-sql
  26. PySpark: How to fillna values in dataframe for specific columns?
  27. How to get Kafka offsets for structured query for manual and reliable offset management?
  28. How to calculate Median in spark sqlContext for column of data type double
  29. Spark: disk I/O on stage boundaries explanation
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment