How to flatmap a nested Dataframe in Spark

by

Spark 2.0+

Dataset.flatMap:

val ds = df.as[(String, String, String, String)]
ds.flatMap { 
  case (x1, x2, x3, x4) => x3.split(",").map((x1, x2, _, x4))
}.toDF

Spark 1.3+.

Use split and explode functions:

val df = Seq(("A", "B", "x,y,z", "D")).toDF("x1", "x2", "x3", "x4")
df.withColumn("x3", explode(split($"x3", ",")))

Spark 1.x

DataFrame.explode (deprecated in Spark 2.x)

df.explode($"x3")(_.getAs[String](0).split(",").map(Tuple1(_)))

More Related Contents:

  1. Spark 2.0 Dataset vs DataFrame
  2. Spark – load CSV file as DataFrame?
  3. How to define partitioning of DataFrame?
  4. Automatically and Elegantly flatten DataFrame in Spark SQL
  5. Spark extracting values from a Row
  6. SparkSQL: How to deal with null values in user defined function?
  7. How to aggregate values into collection after groupBy?
  8. Dropping a nested column from Spark DataFrame
  9. How to read records in JSON format from Kafka using Structured Streaming?
  10. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  11. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  12. Processing multiple files as independent RDD’s in parallel
  13. MatchError while accessing vector column in Spark 2.0
  14. How to save DataFrame directly to Hive?
  15. Spark / Scala: forward fill with last observation
  16. About how to add a new column to an existing DataFrame with random values in Scala
  17. How to use Column.isin with list?
  18. How to compare two dataframe and print columns that are different in scala
  19. Perform a typed join in Scala with Spark Datasets
  20. Apache Spark: Get number of records per partition
  21. Renaming column names of a DataFrame in Spark Scala
  22. DataFrame-ified zipWithIndex
  23. Derive multiple columns from a single column in a Spark DataFrame
  24. How to save a spark DataFrame as csv on disk?
  25. What is going wrong with `unionAll` of Spark `DataFrame`?
  26. How to sort by column in descending order in Spark SQL?
  27. Exploding nested Struct in Spark dataframe
  28. How to get keys and values from MapType column in SparkSQL DataFrame
  29. Why is join not possible after show operator?
  30. Difference between two rows in Spark dataframe

Leave a Comment