Running custom Java class in PySpark

by

In PySpark try the following

from py4j.java_gateway import java_import
java_import(sc._gateway.jvm,"org.foo.module.Foo")

func = sc._gateway.jvm.Foo()
func.fooMethod()

Make sure that you have compiled your Java code into a runnable jar and submit the spark job like so

spark-submit --driver-class-path "name_of_your_jar_file.jar" --jars "name_of_your_jar_file.jar" name_of_your_python_file.py

More Related Contents:

  1. Spark: How to map Python with Scala or Java User Defined Functions?
  2. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  3. Spark Error – Unsupported class file major version
  4. How to flatten a struct in a Spark dataframe?
  5. How to add third-party Java JAR files for use in PySpark
  6. What are the Spark transformations that causes a Shuffle?
  7. PySpark: java.lang.OutofMemoryError: Java heap space
  8. How to set timezone to UTC in Apache Spark?
  9. How to loop my game back to the start
  10. How to find median and quantiles using Spark
  11. Spark Dataframe distinguish columns with duplicated name
  12. java.lang.ClassCastException using lambda expressions in spark job on remote server
  13. How can we JOIN two Spark SQL dataframes using a SQL-esque “LIKE” criterion?
  14. Create a custom Transformer in PySpark ML
  15. Java “Virtual Machine” vs. Python “Interpreter” parlance?
  16. How to reliably guess the encoding between MacRoman, CP1252, Latin1, UTF-8, and ASCII
  17. Why is Apache-Spark – Python so slow locally as compared to pandas?
  18. Using a java library from python
  19. Tuning parameters for implicit pyspark.ml ALS matrix factorization model through pyspark.ml CrossValidator
  20. PySpark DataFrames – way to enumerate without converting to Pandas?
  21. Using UDF ignores condition in when
  22. Concatenating string by rows in pyspark
  23. What is the equivalent to scala.util.Try in pyspark?
  24. How do I convert an array (i.e. list) column to Vector
  25. Add column sum as new column in PySpark dataframe
  26. Difference between SparkContext, JavaSparkContext, SQLContext, and SparkSession?
  27. How to drop all columns with null values in a PySpark DataFrame?
  28. Why does SparkSession execute twice for one action?
  29. How to add a SparkListener from pySpark in Python?
  30. Random numbers generation in PySpark

Leave a Comment