Spark Error – Unsupported class file major version

by

Edit Spark 3.0 supports Java 11, so you’ll need to upgrade

Spark runs on Java 8/11, Scala 2.12, Python 2.7+/3.4+ and R 3.1+. Java 8 prior to version 8u92 support is deprecated as of Spark 3.0.0

Original answer

Until Spark supports Java 11, or higher (which would be hopefully be mentioned at the latest documentation when it is), you have to add in a flag to set your Java version to Java 8.

As of Spark 2.4.x

Spark runs on Java 8, Python 2.7+/3.4+ and R 3.1+. For the Scala API, Spark 2.4.4 uses Scala 2.12. You will need to use a compatible Scala version (2.12.x)

On Mac/Unix, see asdf-java for installing different Javas

On a Mac, I am able to do this in my .bashrc,

export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

On Windows, checkout Chocolately, but seriously just use WSL2 or Docker to run Spark.

You can also set this in spark-env.sh rather than set the variable for your whole profile.

And, of course, this all means you’ll need to install Java 8 in addition to your existing Java 11

More Related Contents:

  1. Pyspark: Exception: Java gateway process exited before sending the driver its port number
  2. Spark: How to map Python with Scala or Java User Defined Functions?
  3. Running custom Java class in PySpark
  4. How to flatten a struct in a Spark dataframe?
  5. How to reliably guess the encoding between MacRoman, CP1252, Latin1, UTF-8, and ASCII
  6. What are the Spark transformations that causes a Shuffle?
  7. PySpark: java.lang.OutofMemoryError: Java heap space
  8. How to set timezone to UTC in Apache Spark?
  9. Android update 17 seems incompatible with external Jars
  10. Custom button not working on mac (ButtonUI)
  11. Installed Java 7 on Mac OS X but Terminal is still using version 6
  12. Does spark predicate pushdown work with JDBC?
  13. Explode in PySpark
  14. Eclipse Installation on macOS
  15. Filtering a Pyspark DataFrame with SQL-like IN clause
  16. Spark union of multiple RDDs
  17. Dividing complex rows of dataframe to simple rows in Pyspark
  18. Spark add new column to dataframe with value from previous row
  19. How to install a specific JDK on Mac OS X?
  20. Pyspark: Replacing value in a column by searching a dictionary
  21. Pivot String column on Pyspark Dataframe
  22. Java or Python for Natural Language Processing [closed]
  23. Reduce a key-value pair into a key-list pair with Apache Spark
  24. Java – How to hook into the Copy and Paste menu on the Mac OS
  25. Maven 3.3.1 ECLIPSE: -Dmaven.multiModuleProjectDirectory system propery is not set
  26. How do I set the driver’s python version in spark?
  27. PySpark in iPython notebook raises Py4JJavaError when using count() and first()
  28. How do I call a UDF on a Spark DataFrame using JAVA?
  29. When to cache a DataFrame?
  30. Using NumPy and Cpython with Jython

Leave a Comment