spark-shell error on Windows – can it be ignored if not using hadoop?

by

tl;dr You’d rather not.

Well, it may be possible, but given you’ve just started your journey to Spark’s land the efforts would not pay off.

Windows has never been a developer-friendly OS to me and whenever I teach people Spark and they use Windows I just take it as granted that we’ll have to go through the winutils.exe setup but many times also how to work on command line.

Please install winutils.exe as follows:

  1. Run cmd as administrator
  2. Download winutils.exe binary from https://github.com/steveloughran/winutils repository (use hadoop-2.7.1 for Spark 2)
  3. Save winutils.exe binary to a directory of your choice, e.g. c:\hadoop\bin
  4. Set HADOOP_HOME to reflect the directory with winutils.exe (without bin), e.g. set HADOOP_HOME=c:\hadoop
  5. Set PATH environment variable to include %HADOOP_HOME%\bin
  6. Create c:\tmp\hive directory
  7. Execute winutils.exe chmod -R 777 \tmp\hive
  8. Open spark-shell and run spark.range(1).show to see a one-row dataset.

More Related Contents:

  1. Spark SQL: apply aggregate functions to a list of columns
  2. Unpivot in spark-sql/pyspark
  3. How to stop INFO messages displaying on spark console?
  4. Spark SQL replacement for MySQL’s GROUP_CONCAT aggregate function
  5. DataFrame join optimization – Broadcast Hash Join
  6. Overwrite specific partitions in spark dataframe write method
  7. Convert date from String to Date format in Dataframes
  8. What is the meaning of partitionColumn, lowerBound, upperBound, numPartitions parameters?
  9. What is the difference between spark checkpoint and persist to a disk
  10. Xml processing in Spark
  11. What are workers, executors, cores in Spark Standalone cluster?
  12. Spark: Reading files using different delimiter than new line
  13. How to fix ‘TypeError: an integer is required (got type bytes)’ error when trying to run pyspark after installing spark 2.4.4
  14. Spark MLlib LDA, how to infer the topics distribution of a new unseen document?
  15. Why does format(“kafka”) fail with “Failed to find data source: kafka.” (even with uber-jar)?
  16. Multiple Spark applications with HiveContext
  17. How to loop through each row of dataFrame in pyspark
  18. How to group by common element in array?
  19. Understanding Spark serialization
  20. Pyspark: Pass multiple columns in UDF
  21. Spark-Obtaining file name in RDDs
  22. DataFrame partitionBy to a single Parquet file (per partition)
  23. Temp table caching with spark-sql
  24. How to assign unique contiguous numbers to elements in a Spark RDD
  25. What is spark.driver.maxResultSize?
  26. Spark + s3 – error – java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
  27. How to get Kafka offsets for structured query for manual and reliable offset management?
  28. When to use Spark DataFrame/Dataset API and when to use plain RDD?
  29. Spark: disk I/O on stage boundaries explanation
  30. spark.ml StringIndexer throws ‘Unseen label’ on fit()

Leave a Comment