SparkSQL: How to deal with null values in user defined function?

This is where Optioncomes in handy: val extractDateAsOptionInt = udf((d: String) => d match { case null => None case s => Some(s.substring(0, 10).filterNot(“-“.toSet).toInt) }) or to make it slightly more secure in general case: import scala.util.Try val extractDateAsOptionInt = udf((d: String) => Try( d.substring(0, 10).filterNot(“-“.toSet).toInt ).toOption) All credit goes to Dmitriy Selivanov who’ve pointed … Read more

How to use .NET reflection to check for nullable reference type

In .NET 6, APIs were added to handle this, see this answer. Prior to this, you need to read the attributes yourself. This appears to work, at least on the types I’ve tested it with. public static bool IsNullable(PropertyInfo property) => IsNullableHelper(property.PropertyType, property.DeclaringType, property.CustomAttributes); public static bool IsNullable(FieldInfo field) => IsNullableHelper(field.FieldType, field.DeclaringType, field.CustomAttributes); public static … Read more

Nullable type is not a nullable type?

According to the MSDN : Calling GetType on a Nullable type causes a boxing operation to be performed when the type is implicitly converted to Object. Therefore GetType always returns a Type object that represents the underlying type, not the Nullable type. When you box a nullable object, only the underlying type is boxed. Again, … Read more