Using UDF ignores condition in when

by

You have to remember that Spark SQL (unlike RDD) is not what-you-see-is-what-you-get. Optimizer / planner is free to schedule operations in the arbitrary order or even repeat stages multiple times.

Python udfs are not applied on a Row basis, but using batch mode. when is not so much ignored, but not used to optimize execution plan:

== Physical Plan ==
*Project [col#0, CASE WHEN isnull(col#0) THEN col#0 ELSE pythonUDF0#21 END AS upper#17]
+- BatchEvalPython [<lambda>(col#0)], [col#0, pythonUDF0#21]
   +- Scan ExistingRDD[col#0]

Therefore function used with udf has to be robust to None inputs, for example:

df.withColumn(
    'upper',
    f.udf(
        lambda x: x.upper() if x is not None else None, 
        StringType()
    )(f.col('col'))
).show()

More Related Contents:

  1. Applying UDFs on GroupedData in PySpark (with functioning python example)
  2. Passing a data frame column and external list to udf under withColumn
  3. How to pass a constant value to Python UDF?
  4. Apache Spark — Assign the result of UDF to multiple dataframe columns
  5. Spark Equivalent of IF Then ELSE
  6. How to add a constant column in a Spark DataFrame?
  7. How to split Vector into columns – using PySpark
  8. Spark Dataframe distinguish columns with duplicated name
  9. How to change dataframe column names in pyspark?
  10. ‘PipelinedRDD’ object has no attribute ‘toDF’ in PySpark
  11. Explode in PySpark
  12. Pyspark: explode json in column to multiple columns
  13. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  14. Dividing complex rows of dataframe to simple rows in Pyspark
  15. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  16. Why is Apache-Spark – Python so slow locally as compared to pandas?
  17. Cast column containing multiple string date formats to DateTime in Spark
  18. How to explode multiple columns of a dataframe in pyspark
  19. Spark add new column to dataframe with value from previous row
  20. PySpark: multiple conditions in when clause
  21. PySpark create new column with mapping from a dict
  22. Pyspark: Replacing value in a column by searching a dictionary
  23. Create Spark DataFrame. Can not infer schema for type
  24. Pivot String column on Pyspark Dataframe
  25. Create single row dataframe from list of list PySpark
  26. Total size of serialized results of 16 tasks (1048.5 MB) is bigger than spark.driver.maxResultSize (1024.0 MB)
  27. Multiple condition filter on dataframe
  28. How to return a “Tuple type” in a UDF in PySpark?
  29. How to connect HBase and Spark using Python?
  30. How to pivot on multiple columns in Spark SQL?

Leave a Comment