How to use a Scala class inside Pyspark

by

Yes it is possible although can be far from trivial. Typically you want a Java (friendly) wrapper so you don’t have to deal with Scala features which cannot be easily expressed using plain Java and as a result don’t play well with Py4J gateway.

Assuming your class is int the package com.example and have Python DataFrame called df

df = ... # Python DataFrame

you’ll have to:

  1. Build a jar using your favorite build tool.

  2. Include it in the driver classpath for example using --driver-class-path argument for PySpark shell / spark-submit. Depending on the exact code you may have to pass it using --jars as well

  3. Extract JVM instance from a Python SparkContext instance:

    jvm = sc._jvm

  4. Extract Scala SQLContext from a SQLContext instance:

    ssqlContext = sqlContext._ssql_ctx

  5. Extract Java DataFrame from the df:

    jdf = df._jdf

  6. Create new instance of SimpleClass:

    simpleObject = jvm.com.example.SimpleClass(ssqlContext, jdf, "v")

  7. Callexe method and wrap the result using Python DataFrame:

    from pyspark.sql import DataFrame

DataFrame(simpleObject.exe(), ssqlContext)

The result should be a valid PySpark DataFrame. You can of course combine all the steps into a single call.

Important: This approach is possible only if Python code is executed solely on the driver. It cannot be used inside Python action or transformation. See How to use Java/Scala function from an action or a transformation? for details.

More Related Contents:

  1. How to use JDBC source to write and read data in (Py)Spark?
  2. Convert pyspark string to date format
  3. How to split Vector into columns – using PySpark
  4. Applying UDFs on GroupedData in PySpark (with functioning python example)
  5. Spark Dataframe distinguish columns with duplicated name
  6. Pyspark: Split multiple array columns into rows
  7. How to change dataframe column names in pyspark?
  8. How to change a dataframe column from String type to Double type in PySpark?
  9. Retrieve top n in each group of a DataFrame in pyspark
  10. Pyspark: explode json in column to multiple columns
  11. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  12. Pyspark ‘NoneType’ object has no attribute ‘_jvm’ error
  13. Filter Pyspark dataframe column with None value
  14. Cast column containing multiple string date formats to DateTime in Spark
  15. How to explode multiple columns of a dataframe in pyspark
  16. PySpark: multiple conditions in when clause
  17. Count number of non-NaN entries in each column of Spark dataframe in PySpark
  18. Create Spark DataFrame. Can not infer schema for type
  19. Load CSV file with PySpark
  20. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  21. Create single row dataframe from list of list PySpark
  22. Cannot find col function in pyspark
  23. What is the best way to remove accents with Apache Spark dataframes in PySpark?
  24. How to zip two array columns in Spark SQL
  25. When to cache a DataFrame?
  26. Multiple condition filter on dataframe
  27. How to return a “Tuple type” in a UDF in PySpark?
  28. How to connect HBase and Spark using Python?
  29. How to pivot on multiple columns in Spark SQL?
  30. Apache Spark — Assign the result of UDF to multiple dataframe columns

Leave a Comment