SparkSQL: How to deal with null values in user defined function?
This is where Optioncomes in handy: val extractDateAsOptionInt = udf((d: String) => d match { case null => None case s => Some(s.substring(0, 10).filterNot(“-“.toSet).toInt) }) or to make it slightly more secure in general case: import scala.util.Try val extractDateAsOptionInt = udf((d: String) => Try( d.substring(0, 10).filterNot(“-“.toSet).toInt ).toOption) All credit goes to Dmitriy Selivanov who’ve pointed … Read more