Exception in thread “main” java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)

by

Make sure Spark is compatible with corresponding Scala version

The error is common when using Scala version 2.12 series with any version of Spark offering Scala 2.11.

You can try using the 2.11 series of Scala with Spark . i.e. 

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.2.0"

As you can see in this dependency spark-core_2.11 is associated with scala version 2.11.

That’s why it’s safer (more compatible) to use %% and avoid hardcoding the version of Scala in Spark dependencies. Let the tool resolve the required Scala version for you automatically as follows:

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.0"

The above declaration will automatically infer the scala version.

More Related Contents:

  1. How to build an Uber JAR (Fat JAR) using SBT within IntelliJ IDEA?
  2. How to force IntelliJ IDEA to reload dependencies from build.sbt after they changed?
  3. Debugging Scala code with simple-build-tool (sbt) and IntelliJ
  4. How to create SBT project with IntelliJ Idea?
  5. “Unknown artifact. Not resolved or indexed” error for scalatest
  6. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  7. Spark2.1.0 incompatible Jackson versions 2.7.6
  8. Run a scala code jar appear NoSuchMethodError:scala.Predef$.refArrayOps
  9. How to store custom objects in Dataset?
  10. Split 1 column into 3 columns in spark scala
  11. Provide schema while reading csv file as a dataframe
  12. How to flatmap a nested Dataframe in Spark
  13. How to set heap size for sbt?
  14. Modify collection inside a Spark RDD foreach
  15. How to write unit tests in Spark 2.0+?
  16. Customize SparkContext using sparkConf.set(..) when using spark-shell
  17. How to add “provided” dependencies back to run/test tasks’ classpath?
  18. Spark: Transpose DataFrame Without Aggregating
  19. Exploding nested Struct in Spark dataframe
  20. How do I iterate RDD’s in apache spark (scala)
  21. How to run jar generated by package (possibly with other jars under lib)?
  22. Spark-Monotonically increasing id not working as expected in dataframe?
  23. How to remove parentheses around records when saveAsTextFile on RDD[(String, Int)]?
  24. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  25. Install sbt on ubuntu [closed]
  26. How to sort an RDD in Scala Spark?
  27. Is there a simple example of how to generate verilog from Chisel3 module?
  28. Difference between two rows in Spark dataframe
  29. Spark UDF with varargs
  30. How to transpose an RDD in Spark

Leave a Comment