Append a column to Data Frame in Apache Spark 1.3

by

It’s been a while since I posted the question and it seems that some other people would like to get an answer as well. Below is what I found.

So the original task was to append a column with row identificators (basically, a sequence 1 to numRows) to any given data frame, so the rows order/presence can be tracked (e.g. when you sample). This can be achieved by something along these lines:

sqlContext.textFile(file).
zipWithIndex().
map(case(d, i)=>i.toString + delimiter + d).
map(_.split(delimiter)).
map(s=>Row.fromSeq(s.toSeq))

Regarding the general case of appending any column to any data frame:

The “closest” to this functionality in Spark API are withColumn and withColumnRenamed. According to Scala docs, the former Returns a new DataFrame by adding a column. In my opinion, this is a bit confusing and incomplete definition. Both of these functions can operate on this data frame only, i.e. given two data frames df1 and df2 with column col:

val df = df1.withColumn("newCol", df1("col") + 1) // -- OK
val df = df1.withColumn("newCol", df2("col") + 1) // -- FAIL

So unless you can manage to transform a column in an existing dataframe to the shape you need, you can’t use withColumn or withColumnRenamed for appending arbitrary columns (standalone or other data frames).

As it was commented above, the workaround solution may be to use a join – this would be pretty messy, although possible – attaching the unique keys like above with zipWithIndex to both data frames or columns might work. Although efficiency is …

It’s clear that appending a column to the data frame is not an easy functionality for distributed environment and there may not be very efficient, neat method for that at all. But I think that it’s still very important to have this core functionality available, even with performance warnings.

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to define partitioning of DataFrame?
  3. How do I detect if a Spark DataFrame has a column
  4. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  5. How to split a dataframe into dataframes with same column values?
  6. Provide schema while reading csv file as a dataframe
  7. Dropping a nested column from Spark DataFrame
  8. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  9. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  10. How to zip two (or more) DataFrame in Spark
  11. Joining Spark dataframes on the key
  12. Apache Spark how to append new column from list/array to Spark dataframe
  13. Create new Dataframe with empty/null field values
  14. Provide schema while reading csv file as a dataframe in Scala Spark
  15. Renaming column names of a DataFrame in Spark Scala
  16. Derive multiple columns from a single column in a Spark DataFrame
  17. Replace missing values with mean – Spark Dataframe
  18. What is going wrong with `unionAll` of Spark `DataFrame`?
  19. Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame
  20. Replace null values in Spark DataFrame
  21. How to get keys and values from MapType column in SparkSQL DataFrame
  22. Add Number of days column to Date Column in same dataframe for Spark Scala App
  23. Spark Dataframe Random UUID changes after every transformation/action
  24. Spark DataFrames when udf functions do not accept large enough input variables
  25. Filter spark DataFrame on string contains
  26. (Why) do we need to call cache or persist on a RDD
  27. Why is “Unable to find encoder for type stored in a Dataset” when creating a dataset of custom case class?
  28. Operate on neighbor elements in RDD in Spark
  29. Spark: long delay between jobs
  30. Spark column string replace when present in other column (row)

Leave a Comment