How to set up Spark on Windows?

by

Steps to install Spark in local mode:

  1. Install Java 7 or later.
    To test java installation is complete, open command prompt type java and hit enter.
    If you receive a message 'Java' is not recognized as an internal or external command. You need to configure your environment variables, JAVA_HOME and PATH to point to the path of jdk.

  2. Download and install Scala.

    Set SCALA_HOME in Control Panel\System and Security\System goto “Adv System settings” and add %SCALA_HOME%\bin in PATH variable in environment variables.

  3. Install Python 2.6 or later from Python Download link.

  4. Download SBT. Install it and set SBT_HOME as an environment variable with value as <<SBT PATH>>.

  5. Download winutils.exe from HortonWorks repo or git repo. Since we don’t have a local Hadoop installation on Windows we have to download winutils.exe and place it in a bin directory under a created Hadoop home directory.
    Set HADOOP_HOME = <<Hadoop home directory>> in environment variable.

  6. We will be using a pre-built Spark package, so choose a Spark pre-built package for Hadoop Spark download. Download and extract it.

    Set SPARK_HOME and add %SPARK_HOME%\bin in PATH variable in environment variables.

  7. Run command: spark-shell

  8. Open http://localhost:4040/ in a browser to see the SparkContext web UI.

More Related Contents:

  1. (null) entry in command string exception in saveAsTextFile() on Pyspark
  2. Why does starting a streaming query lead to “ExitCodeException exitCode=-1073741515”?
  3. How do I get current date/time on the Windows command line in a suitable format for usage in a file/folder name?
  4. How do you get the string length in a batch file?
  5. What are good grep tools for Windows? [closed]
  6. How do you loop through each line in a text file using a windows batch file?
  7. Batch script: how to check for admin rights
  8. Setting a system environment variable from a Windows batch file?
  9. How to schedule a task to run when shutting down windows
  10. Show touch keyboard (TabTip.exe) in Windows 10 Anniversary edition
  11. Create an empty file on the commandline in windows (like the linux touch command)
  12. Where are environment variables stored in the Windows Registry?
  13. How to connect to a docker container from outside the host (same network) [Windows]
  14. Wix: Windows Service sometimes uninstalled when upgrading
  15. How can I run a Windows GUI application on as a service?
  16. How to install OpenJDK 11 on Windows?
  17. SSH to Vagrant box in Windows?
  18. Decode PowerShell output possibly containing non-ASCII Unicode characters into a Python string
  19. How to check if a service is running via batch file and start it, if it is not running?
  20. How can I run a windows batch file but hide the command window?
  21. ‘ssh-keygen’ is not recognized as an internal or external command
  22. How does Windows actually detect LAN (proxy) settings when using Automatic Configuration
  23. Cannot run Maven using `mvn -D` argument within Microsoft Powershell, but works in Command Prompt
  24. How do I increment a folder name using Windows batch?
  25. Using echo y as an automated response to a pcp hostkey prompt
  26. Why does Windows use CR LF?
  27. How to update system PATH variable permanently from cmd?
  28. php-fpm for Windows?
  29. IE Enable/Disable Proxy Settings via Registry
  30. Windows shortcut to run a Git Bash script

Leave a Comment