How to pass a constant value to Python UDF?

by

Everything that is passed to an UDF is interpreted as a column / column name. If you want to pass a literal you have two options:

  1. Pass argument using currying:

    def comparatorUDF(n):
    return udf(lambda c: c == n, BooleanType())

df.where(comparatorUDF("Bonsanto")(col("name")))

    This can be used with an argument of any type as long as it is serializable.

  2. Use a SQL literal and the current implementation:

    from pyspark.sql.functions import lit

df.where(comparatorUDF(col("name"), lit("Bonsanto")))

    This works only with supported types (strings, numerics, booleans). For non-atomic types see How to add a constant column in a Spark DataFrame?

More Related Contents:

  1. Applying UDFs on GroupedData in PySpark (with functioning python example)
  2. Passing a data frame column and external list to udf under withColumn
  3. Using UDF ignores condition in when
  4. Apache Spark — Assign the result of UDF to multiple dataframe columns
  5. Convert pyspark string to date format
  6. How to split Vector into columns – using PySpark
  7. How to use JDBC source to write and read data in (Py)Spark?
  8. Pyspark: Split multiple array columns into rows
  9. How to change a dataframe column from String type to Double type in PySpark?
  10. Retrieve top n in each group of a DataFrame in pyspark
  11. Pyspark: explode json in column to multiple columns
  12. How to use a Scala class inside Pyspark
  13. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How to explode multiple columns of a dataframe in pyspark
  16. PySpark: multiple conditions in when clause
  17. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  18. Create Spark DataFrame. Can not infer schema for type
  19. Load CSV file with PySpark
  20. Create single row dataframe from list of list PySpark
  21. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  22. How to zip two array columns in Spark SQL
  23. Concatenate two PySpark dataframes
  24. How do I convert an array (i.e. list) column to Vector
  25. Filtering DataFrame using the length of a column
  26. Add column sum as new column in PySpark dataframe
  27. How to add suffix and prefix to all columns in python/pyspark dataframe
  28. How to drop all columns with null values in a PySpark DataFrame?
  29. Pyspark changing type of column from date to string
  30. PySpark: compute row maximum of the subset of columns and add to an exisiting dataframe

Leave a Comment