Spark functions vs UDF performance?

by

when would a udf be faster

If you ask about Python UDF the answer is probably never*. Since SQL functions are relatively simple and are not designed for complex tasks it is pretty much impossible compensate the cost of repeated serialization, deserialization and data movement between Python interpreter and JVM.

Does anyone know why this is so

The main reasons are already enumerated above and can be reduced to a simple fact that Spark DataFrame is natively a JVM structure and standard access methods are implemented by simple calls to Java API. UDF from the other hand are implemented in Python and require moving data back and forth.

While PySpark in general requires data movements between JVM and Python, in case of low level RDD API it typically doesn’t require expensive serde activity. Spark SQL adds additional cost of serialization and serialization as well cost of moving data from and to unsafe representation on JVM. The later one is specific to all UDFs (Python, Scala and Java) but the former one is specific to non-native languages.

Unlike UDFs, Spark SQL functions operate directly on JVM and typically are well integrated with both Catalyst and Tungsten. It means these can be optimized in the execution plan and most of the time can benefit from codgen and other Tungsten optimizations. Moreover these can operate on data in its “native” representation.

So in a sense the problem here is that Python UDF has to bring data to the code while SQL expressions go the other way around.

* According to rough estimates PySpark window UDF can beat Scala window function.

More Related Contents:

  1. DataFrame / Dataset groupBy behaviour/optimization
  2. Applying UDFs on GroupedData in PySpark (with functioning python example)
  3. Passing a data frame column and external list to udf under withColumn
  4. Spark UDAF with ArrayType as bufferSchema performance issues
  5. What is the difference between spark.sql.shuffle.partitions and spark.default.parallelism?
  6. Number of partitions in RDD and performance in Spark
  7. Spark Error:expected zero arguments for construction of ClassDict (for numpy.core.multiarray._reconstruct)
  8. Using UDF ignores condition in when
  9. How to pass a constant value to Python UDF?
  10. Apache Spark — Assign the result of UDF to multiple dataframe columns
  11. Apache Spark: map vs mapPartitions?
  12. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  13. How to check if spark dataframe is empty?
  14. How can I pass extra parameters to UDFs in Spark SQL?
  15. Spark performance for Scala vs Python
  16. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  17. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  18. Best way to get the max value in a Spark dataframe column
  19. How to loop through each row of dataFrame in pyspark
  20. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  21. How to set timezone to UTC in Apache Spark?
  22. Cannot find col function in pyspark
  23. How to zip two array columns in Spark SQL
  24. How can I access python variable in Spark SQL?
  25. Concatenate two PySpark dataframes
  26. How do I convert an array (i.e. list) column to Vector
  27. Filtering DataFrame using the length of a column
  28. Add column sum as new column in PySpark dataframe
  29. How to turn off scientific notation in pyspark?
  30. How to drop all columns with null values in a PySpark DataFrame?

Leave a Comment