Running Apache Hadoop 2.1.0 on Windows

by

I have followed following steps to install Hadoop 2.2.0

Steps to build Hadoop bin distribution for Windows

  1. Download and install Microsoft Windows SDK v7.1.

  2. Download and install Unix command-line tool Cygwin.

  3. Download and install Maven 3.1.1.

  4. Download Protocol Buffers 2.5.0 and extract to a folder (say c:\protobuf).

  5. Add Environment Variables JAVA_HOME, M2_HOME and Platform if not added already.
    Note : Variable name Platform is case sensitive. And value will be either x64 or Win32 for building on a 64-bit or 32-bit system.
    Edit Path Variable to add bin directory of Cygwin (say C:\cygwin64\bin), bin directory of Maven (say C:\maven\bin) and installation path of Protocol Buffers (say c:\protobuf).

  6. Download hadoop-2.2.0-src.tar.gz and extract to a folder having short path (say c:\hdfs) to avoid runtime problem due to maximum path length limitation in Windows.

  7. Select Start –> All Programs –> Microsoft Windows SDK v7.1 and open Windows SDK 7.1 Command Prompt. Change directory to Hadoop source code folder (c:\hdfs). Execute mvn package with options -Pdist,native-win -DskipTests -Dtar to create Windows binary tar distribution.

  8. If everything goes well in the previous step, then native distribution hadoop-2.2.0.tar.gz will be created inside C:\hdfs\hadoop-dist\target\hadoop-2.2.0 directory.

Install Hadoop

  1. Extract hadoop-2.2.0.tar.gz to a folder (say c:\hadoop).

  2. Add Environment Variable HADOOP_HOME and edit Path Variable to add bin directory of HADOOP_HOME (say C:\hadoop\bin).

Configure Hadoop

C:\hadoop\etc\hadoop\core-site.xml

<configuration>
        <property>
                <name>fs.defaultFS</name>
                <value>hdfs://localhost:9000</value>
        </property>
</configuration>

C:\hadoop\etc\hadoop\hdfs-site.xml

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
        <property>
                <name>dfs.namenode.name.dir</name>
                <value>file:/hadoop/data/dfs/namenode</value>
        </property>
        <property>
                <name>dfs.datanode.data.dir</name>
                <value>file:/hadoop/data/dfs/datanode</value>
        </property>
</configuration>

C:\hadoop\etc\hadoop\mapred-site.xml

<configuration>
        <property>
           <name>mapreduce.framework.name</name>
           <value>yarn</value>
        </property>
</configuration>

C:\hadoop\etc\hadoop\ yarn-site.xml

<configuration>
        <property>
           <name>yarn.nodemanager.aux-services</name>
           <value>mapreduce_shuffle</value>
        </property>
        <property>
           <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
           <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
</configuration>

Format namenode

For the first time only, namenode needs to be formatted.

C:\Users\abhijitg>cd c:\hadoop\bin 
c:\hadoop\bin>hdfs namenode –format

Start HDFS (Namenode and Datanode)

C:\Users\abhijitg>cd c:\hadoop\sbin
c:\hadoop\sbin>start-dfs

Start MapReduce aka YARN (Resource Manager and Node Manager)

C:\Users\abhijitg>cd c:\hadoop\sbin
c:\hadoop\sbin>start-yarn
starting yarn daemons

Total four separate Command Prompt windows will be opened automatically to run Namenode, Datanode, Resource Manager, Node Manager

Reference : Build, Install, Configure and Run Apache Hadoop 2.2.0 in Microsoft Windows OS

More Related Contents:

  1. Using UTF-8 Encoding (CHCP 65001) in Command Prompt / Windows Powershell (Windows 10)
  2. Qt static linking and deployment
  3. How to install pkg config in windows?
  4. How to generate an import library (LIB-file) from a DLL?
  5. What does %~d0 mean in a Windows batch file?
  6. Possible to change where Android Virtual Devices are saved?
  7. How to run a PowerShell script from a batch file
  8. How do I set the version information for an existing .exe, .dll?
  9. How can I determine whether a specific file is open in Windows? [closed]
  10. How can I run a program from a batch file without leaving the console open after the program starts?
  11. Groovy Shell warning “Could not open/create prefs root node …”
  12. How do I find out which process is listening on a TCP or UDP port on Windows? [closed]
  13. Batch files: List all files in a directory with relative paths
  14. python3 print unicode to windows xp console encode cp437
  15. How to run an awk commands in Windows?
  16. What is the smallest possible Windows (PE) executable?
  17. env: bash\r: No such file or directory [duplicate]
  18. Launch an external application from node.js
  19. Docker – failed to compute cache key: not found – runs fine in Visual Studio
  20. Determine Intel HD Graphics Card Version Via WinAPI
  21. Set window position of an application on Windows command line
  22. Programmatically select multiple files in windows explorer
  23. How do I mount a Docker volume while using a Windows host?
  24. Where does Microsoft Windows’ 7 POSIX implementation currently stand?
  25. How to change Screen buffer size in Windows Command Prompt from batch script
  26. How can I run Cygwin Bash Shell from within Emacs?
  27. Check if a Win32 thread is running or in a suspended state
  28. Windows Script Host Error: Invalid character – Code:800A03F6 – Source: Microsoft JScript compilation error
  29. Accessing a single file with multiple threads
  30. PsExec Throws Error Messages, but works without any problems

Leave a Comment