Provide schema while reading csv file as a dataframe in Scala Spark

by

Try the below code, you need not specify the schema. When you give inferSchema as true it should take it from your csv file.

val pagecount = sqlContext.read.format("csv")
  .option("delimiter"," ").option("quote","")
  .option("header", "true")
  .option("inferSchema", "true")
  .load("dbfs:/databricks-datasets/wikipedia-datasets/data-001/pagecounts/sample/pagecounts-20151124-170000")

If you want to manually specify the schema, you can do it as below:

import org.apache.spark.sql.types._

val customSchema = StructType(Array(
  StructField("project", StringType, true),
  StructField("article", StringType, true),
  StructField("requests", IntegerType, true),
  StructField("bytes_served", DoubleType, true))
)

val pagecount = sqlContext.read.format("csv")
  .option("delimiter"," ").option("quote","")
  .option("header", "true")
  .schema(customSchema)
  .load("dbfs:/databricks-datasets/wikipedia-datasets/data-001/pagecounts/sample/pagecounts-20151124-170000")

More Related Contents:

  1. Provide schema while reading csv file as a dataframe
  2. How to query JSON data column using Spark DataFrames?
  3. How to define partitioning of DataFrame?
  4. How do I detect if a Spark DataFrame has a column
  5. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  6. How to split a dataframe into dataframes with same column values?
  7. Dropping a nested column from Spark DataFrame
  8. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  9. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  10. How to zip two (or more) DataFrame in Spark
  11. Joining Spark dataframes on the key
  12. Apache Spark how to append new column from list/array to Spark dataframe
  13. Create new Dataframe with empty/null field values
  14. Renaming column names of a DataFrame in Spark Scala
  15. Derive multiple columns from a single column in a Spark DataFrame
  16. Replace missing values with mean – Spark Dataframe
  17. What is going wrong with `unionAll` of Spark `DataFrame`?
  18. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  19. How to get keys and values from MapType column in SparkSQL DataFrame
  20. Spark DataFrames when udf functions do not accept large enough input variables
  21. Filter spark DataFrame on string contains
  22. How to select the first row of each group?
  23. Automatically and Elegantly flatten DataFrame in Spark SQL
  24. Spark extracting values from a Row
  25. Append a column to Data Frame in Apache Spark 1.3
  26. Processing multiple files as independent RDD’s in parallel
  27. MatchError while accessing vector column in Spark 2.0
  28. Spark / Scala: forward fill with last observation
  29. Apache Spark: Get number of records per partition
  30. How to create a Dataset of Maps?

Leave a Comment