SparkSQL: How to deal with null values in user defined function?

by

This is where Optioncomes in handy:

val extractDateAsOptionInt = udf((d: String) => d match {
  case null => None
  case s => Some(s.substring(0, 10).filterNot("-".toSet).toInt)
})

or to make it slightly more secure in general case:

import scala.util.Try

val extractDateAsOptionInt = udf((d: String) => Try(
  d.substring(0, 10).filterNot("-".toSet).toInt
).toOption)

All credit goes to Dmitriy Selivanov who’ve pointed out this solution as a (missing?) edit here.

Alternative is to handle null outside the UDF:

import org.apache.spark.sql.functions.{lit, when}
import org.apache.spark.sql.types.IntegerType

val extractDateAsInt = udf(
   (d: String) => d.substring(0, 10).filterNot("-".toSet).toInt
)

df.withColumn("y",
  when($"x".isNull, lit(null))
    .otherwise(extractDateAsInt($"x"))
    .cast(IntegerType)
)

More Related Contents:

  1. How to define and use a User-Defined Aggregate Function in Spark SQL?
  2. How can I pass extra parameters to UDFs in Spark SQL?
  3. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  4. About how to add a new column to an existing DataFrame with random values in Scala
  5. Spark UDAF with ArrayType as bufferSchema performance issues
  6. Derive multiple columns from a single column in a Spark DataFrame
  7. How to convert rdd object to dataframe in spark
  8. Spark 2.0 Dataset vs DataFrame
  9. Automatically and Elegantly flatten DataFrame in Spark SQL
  10. Spark extracting values from a Row
  11. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  12. Provide schema while reading csv file as a dataframe
  13. Processing multiple files as independent RDD’s in parallel
  14. How to flatmap a nested Dataframe in Spark
  15. MatchError while accessing vector column in Spark 2.0
  16. How to save DataFrame directly to Hive?
  17. Spark / Scala: forward fill with last observation
  18. Joining Spark dataframes on the key
  19. How to use Column.isin with list?
  20. How to write unit tests in Spark 2.0+?
  21. How to compare two dataframe and print columns that are different in scala
  22. Apache Spark: Get number of records per partition
  23. Renaming column names of a DataFrame in Spark Scala
  24. How to save a spark DataFrame as csv on disk?
  25. What is going wrong with `unionAll` of Spark `DataFrame`?
  26. How to sort by column in descending order in Spark SQL?
  27. Exploding nested Struct in Spark dataframe
  28. How to get keys and values from MapType column in SparkSQL DataFrame
  29. How to create DataFrame from Scala’s List of Iterables?
  30. How to create a Dataset of Maps?

Leave a Comment