Spark2.1.0 incompatible Jackson versions 2.7.6

by

Spark 2.1.0 contains com.fasterxml.jackson.core as transitive dependency. So, we do not need to include then in libraryDependencies.

But if you want to add a different com.fasterxml.jackson.core dependencies’ version then you have to override them. Like this:

name := "testSpark2"

version := "1.0"

scalaVersion := "2.11.8"


dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7"
dependencyOverrides += "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7"

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mllib_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-repl_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-flume_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-sql_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-network-shuffle_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-hive_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-streaming-flume-assembly_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-mesos_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-graphx_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-catalyst_2.11" % "2.1.0"
libraryDependencies += "org.apache.spark" % "spark-launcher_2.11" % "2.1.0"

So, change your build.sbt like the one above and it will work as expected.

I hope it helps!

More Related Contents:

  1. Exception in thread “main” java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)
  2. How to prevent java.lang.OutOfMemoryError: PermGen space at Scala compilation?
  3. How to bucket the range of values from a column and count how many values fall into each interval in scala?
  4. How to query JSON data column using Spark DataFrames?
  5. How to build an Uber JAR (Fat JAR) using SBT within IntelliJ IDEA?
  6. How do I detect if a Spark DataFrame has a column
  7. How to avoid duplicate columns after join?
  8. Flattening Rows in Spark
  9. Spark unionAll multiple dataframes
  10. Spark performance for Scala vs Python
  11. How to specify JVM maximum heap size “-Xmx” for running an application with “run” action in SBT?
  12. How to zip two (or more) DataFrame in Spark
  13. How to force DataFrame evaluation in Spark
  14. Stackoverflow due to long RDD Lineage
  15. call of distinct and map together throws NPE in spark library
  16. How to get path to the uploaded file
  17. how to filter out a null value from spark dataframe
  18. Is there a reason not to use SparkContext.getOrCreate when writing a spark job?
  19. What are the various join types in Spark?
  20. Replace missing values with mean – Spark Dataframe
  21. How to calculate the size of dataframe in bytes in Spark?
  22. Why does partition parameter of SparkContext.textFile not take effect?
  23. Spark Scala list folders in directory
  24. Spark-Monotonically increasing id not working as expected in dataframe?
  25. How to remove parentheses around records when saveAsTextFile on RDD[(String, Int)]?
  26. Reading in multiple files compressed in tar.gz archive into Spark [duplicate]
  27. How to print the contents of RDD?
  28. Coalesce reduces parallelism of entire stage (spark)
  29. Difference between two rows in Spark dataframe
  30. SBT stop run without exiting

Leave a Comment