Apache Spark how to append new column from list/array to Spark dataframe

by

You could do it like this:

import org.apache.spark.sql.Row
import org.apache.spark.sql.types._    

// create rdd from the list
val rdd = sc.parallelize(List(4,5,10,7,2))
// rdd: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[31] at parallelize at <console>:28

// zip the data frame with rdd
val rdd_new = df.rdd.zip(rdd).map(r => Row.fromSeq(r._1.toSeq ++ Seq(r._2)))
// rdd_new: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row] = MapPartitionsRDD[33] at map at <console>:32

// create a new data frame from the rdd_new with modified schema
spark.createDataFrame(rdd_new, df.schema.add("new_col", IntegerType)).show
+----+-------+
|row1|new_col|
+----+-------+
|   a|      4|
|   b|      5|
|   c|     10|
|   d|      7|
|   e|      2|
+----+-------+

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to define partitioning of DataFrame?
  3. How do I detect if a Spark DataFrame has a column
  4. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  5. How to split a dataframe into dataframes with same column values?
  6. Provide schema while reading csv file as a dataframe
  7. Dropping a nested column from Spark DataFrame
  8. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  9. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  10. How to zip two (or more) DataFrame in Spark
  11. Joining Spark dataframes on the key
  12. Create new Dataframe with empty/null field values
  13. Provide schema while reading csv file as a dataframe in Scala Spark
  14. Renaming column names of a DataFrame in Spark Scala
  15. Derive multiple columns from a single column in a Spark DataFrame
  16. Replace missing values with mean – Spark Dataframe
  17. What is going wrong with `unionAll` of Spark `DataFrame`?
  18. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  19. How to get keys and values from MapType column in SparkSQL DataFrame
  20. Spark DataFrames when udf functions do not accept large enough input variables
  21. Filter spark DataFrame on string contains
  22. How to define schema for custom type in Spark SQL?
  23. Spark unionAll multiple dataframes
  24. How can I pass extra parameters to UDFs in Spark SQL?
  25. How to create an empty DataFrame with a specified schema?
  26. How to define a custom aggregation function to sum a column of Vectors?
  27. Better way to convert a string field into timestamp in Spark
  28. Replace null values in Spark DataFrame
  29. Calculate Cosine Similarity Spark Dataframe
  30. Difference between two rows in Spark dataframe

Leave a Comment