Add Jar to standalone pyspark

by

2021-01-19 Updated

There are many approaches here (setting ENV vars, adding to $SPARK_HOME/conf/spark-defaults.conf, etc…) other answers already cover these. I wanted to add an answer for those specifically wanting to do this from within a Python Script or Jupyter Notebook.

When you create the Spark session you can add a .config() that pulls in the specific Jar file (in my case I wanted the Kafka package loaded):

spark = SparkSession.builder.appName('my_awesome')\
    .config('spark.jars.packages', 'org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1')\
    .getOrCreate()

Using this line of code I didn’t need to do anything else (no ENVs or conf file changes).

  • Note 1: The JAR file will dynamically download, you don’t need to manually download it.
  • Note 2: Make sure the versions match what you want, so in the example above my Spark version is 3.0.1 so I have :3.0.1 at the end.

More Related Contents:

  1. Calling Java/Scala function from a task
  2. Efficient string matching in Apache Spark
  3. Load CSV file with Spark
  4. How to link PyCharm with PySpark?
  5. How to turn off INFO logging in Spark?
  6. How do I add a new column to a Spark DataFrame (using PySpark)?
  7. Count number of non-NaN entries in each column of Spark dataframe with Pyspark
  8. importing pyspark in python shell
  9. collect_list by preserving order based on another variable
  10. Rename nested field in spark dataframe
  11. Spark DataFrame: Computing row-wise mean (or any aggregate operation)
  12. Passing a data frame column and external list to udf under withColumn
  13. How to determine if object is a valid key-value pair in PySpark
  14. PySpark converting a column of type ‘map’ to multiple columns in a dataframe
  15. Best way to get the max value in a Spark dataframe column
  16. How to map features from the output of a VectorAssembler back to the column names in Spark ML?
  17. Apache Spark Python Cosine Similarity over DataFrames
  18. Comparing columns in Pyspark
  19. Pyspark: Parse a column of json strings
  20. Reshaping/Pivoting data in Spark RDD and/or Spark DataFrames
  21. Cannot find col function in pyspark
  22. PySpark: match the values of a DataFrame column against another DataFrame column
  23. Spark mllib predicting weird number or NaN
  24. When to cache a DataFrame?
  25. environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON
  26. PySpark Throwing error Method __getnewargs__([]) does not exist
  27. Pyspark 2.4.0, read avro from kafka with read stream – Python
  28. PySpark Evaluation
  29. Apache Spark — Assign the result of UDF to multiple dataframe columns
  30. What is the difference between spark-submit and pyspark?

Leave a Comment