How to flatten a struct in a Spark dataframe?

by

This should work in Spark 1.6 or later:

df.select(df.col("data.*"))

or

df.select(df.col("data.id"), df.col("data.keyNote"), df.col("data.details"))

More Related Contents:

  1. How to set timezone to UTC in Apache Spark?
  2. How to handle this scenario in spark? [closed]
  3. How to do aggregate functions, may columns and extract back
  4. Spark: How to map Python with Scala or Java User Defined Functions?
  5. How to create SparkSession with Hive support (fails with “Hive classes are not found”)?
  6. Spark sql how to explode without losing null values
  7. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  8. Spark Error – Unsupported class file major version
  9. Spark Strutured Streaming automatically converts timestamp to local time
  10. spark broadcast variable Map giving null value
  11. PySpark: java.lang.OutofMemoryError: Java heap space
  12. Change output filename prefix for DataFrame.write()
  13. How do I call a UDF on a Spark DataFrame using JAVA?
  14. Retain keys with null values while writing JSON in spark
  15. How to use Column.isin in Java?
  16. Why does SparkSession execute twice for one action?
  17. Running custom Java class in PySpark
  18. How to run independent transformations in parallel using PySpark?
  19. Spark Window Functions – rangeBetween dates
  20. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  21. Spark add new column to dataframe with value from previous row
  22. Why does Spark think this is a cross / Cartesian join
  23. Pyspark: Replacing value in a column by searching a dictionary
  24. Pivot String column on Pyspark Dataframe
  25. Spark spark-submit –jars arguments wants comma list, how to declare a directory of jars?
  26. How to get element by Index in Spark RDD (Java)
  27. How do I convert an array (i.e. list) column to Vector
  28. Add column sum as new column in PySpark dataframe
  29. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  30. How to drop all columns with null values in a PySpark DataFrame?

Leave a Comment