NoClassDefFoundError com.apache.hadoop.fs.FSDataInputStream when execute spark-shell

by

The “without Hadoop” in the Spark’s build name is misleading: it means the build is not tied to a specific Hadoop distribution, not that it is meant to run without it: the user should indicate where to find Hadoop (see https://spark.apache.org/docs/latest/hadoop-provided.html)

One clean way to fix this issue is to:

  1. Obtain Hadoop Windows binaries. Ideally build them, but this is painful (for some hints see: Hadoop on Windows Building/ Installation Error). Otherwise Google some up, for instance currently you can download 2.6.0 from here: http://www.barik.net/archive/2015/01/19/172716/
  2. Create a spark-env.cmd file looking like this (modify Hadoop path to match your installation):

    @echo off
    set HADOOP_HOME=D:\Utils\hadoop-2.7.1
    set PATH=%HADOOP_HOME%\bin;%PATH%
    set SPARK_DIST_CLASSPATH=<paste here the output of %HADOOP_HOME%\bin\hadoop classpath>
  3. Put this spark-env.cmd either in a conf folder located at the same level as your Spark base folder (which may look weird), or in a folder indicated by the SPARK_CONF_DIR environment variable.

More Related Contents:

  1. Find maximum row per group in Spark DataFrame
  2. Partitioning in spark while reading from RDBMS via JDBC
  3. What is the difference between map and flatMap and a good use case for each?
  4. Best Practice to launch Spark Applications via Web Application?
  5. How to save/insert each DStream into a permanent table
  6. How DAG works under the covers in RDD?
  7. Spark yarn cluster vs client – how to choose which one to use?
  8. Spark on YARN + Secured hbase
  9. Spark load data and add filename as dataframe column
  10. PySpark: how to resample frequencies
  11. Any performance issues forcing eager evaluation using count in spark?
  12. Explode array data into rows in spark [duplicate]
  13. Debugging “Managed memory leak detected” in Spark 1.6.0
  14. Spark: how to get the number of written rows?
  15. ALS model – how to generate full_u * v^t * v?
  16. PySpark – get row number for each row in a group
  17. Why are Spark Parquet files for an aggregate larger than the original?
  18. Determining optimal number of Spark partitions based on workers, cores and DataFrame size
  19. Apache spark dealing with case statements
  20. pyspark: count distinct over a window
  21. Mind blown: RDD.zip() method
  22. How spark read a large file (petabyte) when file can not be fit in spark’s main memory
  23. Serialize a custom transformer using python to be used within a Pyspark ML pipeline
  24. Not able to cat dbfs file in databricks community edition cluster. FileNotFoundError: [Errno 2] No such file or directory:
  25. What do the numbers on the progress bar mean in spark-shell?
  26. pyspark: rolling average using timeseries data
  27. How to improve broadcast Join speed with between condition in Spark
  28. Spark ALS predictAll returns empty
  29. Spark SQL referencing attributes of UDT
  30. How to add a SparkListener from pySpark in Python?

Leave a Comment