Apache Spark, add an “CASE WHEN … ELSE …” calculated column to an existing DataFrame

by

In the upcoming SPARK 1.4.0 release (should be released in the next couple of days). You can use the when/otherwise syntax:

// Create the dataframe
val df = Seq("Red", "Green", "Blue").map(Tuple1.apply).toDF("color")

// Use when/otherwise syntax
val df1 = df.withColumn("Green_Ind", when($"color" === "Green", 1).otherwise(0))

If you are using SPARK 1.3.0 you can chose to use a UDF:

// Define the UDF
val isGreen = udf((color: String) => {
  if (color == "Green") 1
  else 0
})
val df2 = df.withColumn("Green_Ind", isGreen($"color"))

More Related Contents:

  1. How to query JSON data column using Spark DataFrames?
  2. How to define partitioning of DataFrame?
  3. How do I detect if a Spark DataFrame has a column
  4. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  5. How to split a dataframe into dataframes with same column values?
  6. Provide schema while reading csv file as a dataframe
  7. Dropping a nested column from Spark DataFrame
  8. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  9. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  10. How to zip two (or more) DataFrame in Spark
  11. Joining Spark dataframes on the key
  12. Apache Spark how to append new column from list/array to Spark dataframe
  13. Create new Dataframe with empty/null field values
  14. Provide schema while reading csv file as a dataframe in Scala Spark
  15. Renaming column names of a DataFrame in Spark Scala
  16. Derive multiple columns from a single column in a Spark DataFrame
  17. Replace missing values with mean – Spark Dataframe
  18. What is going wrong with `unionAll` of Spark `DataFrame`?
  19. How to get keys and values from MapType column in SparkSQL DataFrame
  20. Spark DataFrames when udf functions do not accept large enough input variables
  21. Filter spark DataFrame on string contains
  22. Spark – load CSV file as DataFrame?
  23. Filling gaps in timeseries Spark
  24. How to use COGROUP for large datasets
  25. Spark Dataframes UPSERT to Postgres Table
  26. What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
  27. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  28. Spark: Add column to dataframe conditionally
  29. Why is join not possible after show operator?
  30. How to create a Dataset of Maps?

Leave a Comment