How to create DataFrame from Scala’s List of Iterables?

by

Thats what spark implicits object is for. It allows you to convert your common scala collection types into DataFrame / DataSet / RDD.
Here is an example with Spark 2.0 but it exists in older versions too

import org.apache.spark.sql.SparkSession
val values = List(1,2,3,4,5)

val spark = SparkSession.builder().master("local").getOrCreate()
import spark.implicits._
val df = values.toDF()

Edit: Just realised you were after 2d list. Here is something I tried on spark-shell. I converted a 2d List to List of Tuples and used implicit conversion to DataFrame: 

val values = List(List("1", "One") ,List("2", "Two") ,List("3", "Three"),List("4","4")).map(x =>(x(0), x(1)))
import spark.implicits._
val df = values.toDF

Edit2: The original question by MTT was How to create spark dataframe from a scala list for a 2d list for which this is a correct answer. The original question is https://stackoverflow.com/revisions/38063195/1
The question was later changed to match an accepted answer. Adding this edit so that if someone else looking for something similar to the original question can find it.

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to convert rdd object to dataframe in spark
  3. How to define and use a User-Defined Aggregate Function in Spark SQL?
  4. Automatically and Elegantly flatten DataFrame in Spark SQL
  5. Spark extracting values from a Row
  6. How do I detect if a Spark DataFrame has a column
  7. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  8. Provide schema while reading csv file as a dataframe
  9. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  10. Processing multiple files as independent RDD’s in parallel
  11. How to flatmap a nested Dataframe in Spark
  12. MatchError while accessing vector column in Spark 2.0
  13. How to save DataFrame directly to Hive?
  14. Spark / Scala: forward fill with last observation
  15. About how to add a new column to an existing DataFrame with random values in Scala
  16. Joining Spark dataframes on the key
  17. How to use Column.isin with list?
  18. How to write unit tests in Spark 2.0+?
  19. How to compare two dataframe and print columns that are different in scala
  20. Apache Spark: Get number of records per partition
  21. Renaming column names of a DataFrame in Spark Scala
  22. Derive multiple columns from a single column in a Spark DataFrame
  23. How to save a spark DataFrame as csv on disk?
  24. What is going wrong with `unionAll` of Spark `DataFrame`?
  25. How to sort by column in descending order in Spark SQL?
  26. Exploding nested Struct in Spark dataframe
  27. How to get keys and values from MapType column in SparkSQL DataFrame
  28. How to create a Dataset of Maps?
  29. Spark DataFrames when udf functions do not accept large enough input variables
  30. Filter spark DataFrame on string contains

Leave a Comment