Spark – load CSV file as DataFrame?

by

spark-csv is part of core Spark functionality and doesn’t require a separate library.
So you could just do for example 

df = spark.read.format("csv").option("header", "true").load("csvfile.csv")

In scala,(this works for any format-in delimiter mention “,” for csv, “\t” for tsv etc)

val df = sqlContext.read.format("com.databricks.spark.csv")
.option("delimiter", ",")
.load("csvfile.csv")

More Related Contents:

  1. Write to multiple outputs by key Spark – one Spark job
  2. Apache Spark: Get number of records per partition
  3. How to define partitioning of DataFrame?
  4. Encoder error while trying to map dataframe row to updated row
  5. Spark extracting values from a Row
  6. Spark Scala: How to convert Dataframe[vector] to DataFrame[f1:Double, …, fn: Double)]
  7. How can I change column types in Spark SQL’s DataFrame?
  8. SparkSQL: How to deal with null values in user defined function?
  9. How to aggregate values into collection after groupBy?
  10. Dropping a nested column from Spark DataFrame
  11. How to read records in JSON format from Kafka using Structured Streaming?
  12. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  13. MatchError while accessing vector column in Spark 2.0
  14. Spark DataFrame: does groupBy after orderBy maintain that order?
  15. Filling gaps in timeseries Spark
  16. Spark UDAF with ArrayType as bufferSchema performance issues
  17. Apache Spark how to append new column from list/array to Spark dataframe
  18. How to get ID of a map task in Spark?
  19. How to use COGROUP for large datasets
  20. What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
  21. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  22. Create new Dataframe with empty/null field values
  23. Perform a typed join in Scala with Spark Datasets
  24. How to Define Custom partitioner for Spark RDDs of equally sized partition where each partition has equal number of elements?
  25. DataFrame-ified zipWithIndex
  26. Spark: Add column to dataframe conditionally
  27. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  28. Calculate Cosine Similarity Spark Dataframe
  29. Why is join not possible after show operator?
  30. Difference between two rows in Spark dataframe

Leave a Comment