Spark: Add column to dataframe conditionally

by

Try withColumn with the function when as follows:

val sqlContext = new SQLContext(sc)
import sqlContext.implicits._ // for `toDF` and $""
import org.apache.spark.sql.functions._ // for `when`

val df = sc.parallelize(Seq((4, "blah", 2), (2, "", 3), (56, "foo", 3), (100, null, 5)))
    .toDF("A", "B", "C")

val newDf = df.withColumn("D", when($"B".isNull or $"B" === "", 0).otherwise(1))

newDf.show() shows

+---+----+---+---+
|  A|   B|  C|  D|
+---+----+---+---+
|  4|blah|  2|  1|
|  2|    |  3|  0|
| 56| foo|  3|  1|
|100|null|  5|  0|
+---+----+---+---+

I added the (100, null, 5) row for testing the isNull case.

I tried this code with Spark 1.6.0 but as commented in the code of when, it works on the versions after 1.4.0.

More Related Contents:

  1. Spark 2.0 Dataset vs DataFrame
  2. Spark – load CSV file as DataFrame?
  3. How to define partitioning of DataFrame?
  4. Automatically and Elegantly flatten DataFrame in Spark SQL
  5. Spark extracting values from a Row
  6. SparkSQL: How to deal with null values in user defined function?
  7. How to aggregate values into collection after groupBy?
  8. Dropping a nested column from Spark DataFrame
  9. How to read records in JSON format from Kafka using Structured Streaming?
  10. Spark Dataframe :How to add a index Column : Aka Distributed Data Index
  11. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  12. Processing multiple files as independent RDD’s in parallel
  13. MatchError while accessing vector column in Spark 2.0
  14. Spark / Scala: forward fill with last observation
  15. Filling gaps in timeseries Spark
  16. How to use COGROUP for large datasets
  17. What are possible reasons for receiving TimeoutException: Futures timed out after [n seconds] when working with Spark [duplicate]
  18. Why does join fail with “java.util.concurrent.TimeoutException: Futures timed out after [300 seconds]”?
  19. How to compare two dataframe and print columns that are different in scala
  20. Create new Dataframe with empty/null field values
  21. Perform a typed join in Scala with Spark Datasets
  22. Apache Spark: Get number of records per partition
  23. DataFrame-ified zipWithIndex
  24. Derive multiple columns from a single column in a Spark DataFrame
  25. How to save a spark DataFrame as csv on disk?
  26. What is going wrong with `unionAll` of Spark `DataFrame`?
  27. How to sort by column in descending order in Spark SQL?
  28. How to get keys and values from MapType column in SparkSQL DataFrame
  29. Why is join not possible after show operator?
  30. Difference between two rows in Spark dataframe

Leave a Comment