Explode array data into rows in spark [duplicate]

by

The explode function should get that done.

pyspark version:

>>> df = spark.createDataFrame([(1, "A", [1,2,3]), (2, "B", [3,5])],["col1", "col2", "col3"])
>>> from pyspark.sql.functions import explode
>>> df.withColumn("col3", explode(df.col3)).show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   1|   A|   1|
|   1|   A|   2|
|   1|   A|   3|
|   2|   B|   3|
|   2|   B|   5|
+----+----+----+

Scala version

scala> val df = Seq((1, "A", Seq(1,2,3)), (2, "B", Seq(3,5))).toDF("col1", "col2", "col3")
df: org.apache.spark.sql.DataFrame = [col1: int, col2: string ... 1 more field]

scala> df.withColumn("col3", explode($"col3")).show()
+----+----+----+
|col1|col2|col3|
+----+----+----+
|   1|   A|   1|
|   1|   A|   2|
|   1|   A|   3|
|   2|   B|   3|
|   2|   B|   5|
+----+----+----+

More Related Contents:

  1. Finding duplicates from large data set using Apache Spark
  2. Find maximum row per group in Spark DataFrame
  3. How do I split an RDD into two or more RDDs?
  4. How to access element of a VectorUDT column in a Spark DataFrame?
  5. Spark Dataframe validating column names for parquet writes
  6. How to improve performance for slow Spark jobs using DataFrame and JDBC connection?
  7. TypeError: Column is not iterable – How to iterate over ArrayType()?
  8. Python Spark Cumulative Sum by Group Using DataFrame
  9. Link Spark with iPython Notebook
  10. How to save/insert each DStream into a permanent table
  11. Pyspark : forward fill with last observation for a DataFrame
  12. Spark load data and add filename as dataframe column
  13. PySpark: how to resample frequencies
  14. Why does Spark think this is a cross / Cartesian join
  15. Apache Spark: What is the equivalent implementation of RDD.groupByKey() using RDD.aggregateByKey()?
  16. PySpark – get row number for each row in a group
  17. How to find count of Null and Nan values for each column in a PySpark dataframe efficiently?
  18. Save ML model for future usage
  19. Apache spark dealing with case statements
  20. pyspark: count distinct over a window
  21. pyspark: Efficiently have partitionBy write to same number of total partitions as original table
  22. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  23. AWS EMR – ModuleNotFoundError: No module named ‘pyarrow’
  24. Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:
  25. PySpark: How to fillna values in dataframe for specific columns?
  26. pyspark: rolling average using timeseries data
  27. Spark ALS predictAll returns empty
  28. Why is the fold action necessary in Spark?
  29. How to add a SparkListener from pySpark in Python?
  30. PySpark error: AttributeError: ‘NoneType’ object has no attribute ‘_jvm’

Leave a Comment