How can I pass extra parameters to UDFs in Spark SQL?

by

Just use a little bit of currying:

def convertDateFunc(resolution: DateResolutionType) = udf((x:String) => 
  SparkDateTimeConverter.convertDate(x, resolution))

and use it as follows:

case FieldDataType.Date => convertDateFunc(resolution(i))(allCols(i))

On a side note you should take a look at sql.functions.trunc and sql.functions.date_format. These should at least part of the job without using UDFs at all.

Note:

In Spark 2.2 or later you can use typedLit function:

import org.apache.spark.sql.functions.typedLit

which support a wider range of literals like Seq or Map.

More Related Contents:

  1. How to define and use a User-Defined Aggregate Function in Spark SQL?
  2. SparkSQL: How to deal with null values in user defined function?
  3. Defining a UDF that accepts an Array of objects in a Spark DataFrame?
  4. About how to add a new column to an existing DataFrame with random values in Scala
  5. Spark UDAF with ArrayType as bufferSchema performance issues
  6. Derive multiple columns from a single column in a Spark DataFrame
  7. How to convert rdd object to dataframe in spark
  8. Automatically and Elegantly flatten DataFrame in Spark SQL
  9. Spark extracting values from a Row
  10. Caused by: java.lang.NullPointerException at org.apache.spark.sql.Dataset
  11. Provide schema while reading csv file as a dataframe
  12. How to flatmap a nested Dataframe in Spark
  13. How to save DataFrame directly to Hive?
  14. Joining Spark dataframes on the key
  15. How to use Column.isin with list?
  16. How to write unit tests in Spark 2.0+?
  17. how to filter out a null value from spark dataframe
  18. How to compare two dataframe and print columns that are different in scala
  19. What are the various join types in Spark?
  20. Renaming column names of a DataFrame in Spark Scala
  21. Replace missing values with mean – Spark Dataframe
  22. What is going wrong with `unionAll` of Spark `DataFrame`?
  23. How to calculate the size of dataframe in bytes in Spark?
  24. Exploding nested Struct in Spark dataframe
  25. How to get keys and values from MapType column in SparkSQL DataFrame
  26. Spark-Monotonically increasing id not working as expected in dataframe?
  27. Spark column string replace when present in other column (row)
  28. How to create a Dataset of Maps?
  29. Spark DataFrames when udf functions do not accept large enough input variables
  30. Filter spark DataFrame on string contains

Leave a Comment