You should also properly configure the maximum memory allocations for MapReduce. From this HortonWorks tutorial: […] Each machine in our cluster has 48 GB of RAM. Some of this RAM should be >reserved for Operating System usage. On each node, we’ll assign 40 GB RAM for >YARN to use and keep 8 GB for the … Read more
Partitioning data is often used for distributing load horizontally, this has performance benefit, and helps in organizing data in a logical fashion. Example: if we are dealing with a large employee table and often run queries with WHERE clauses that restrict the results to a particular country or department . For a faster query response … Read more
Instead of doing the file merging on your own, you can delegate the entire merging of the reduce output files by calling: hadoop fs -getmerge /output/dir/on/hdfs/ /desired/local/output/file.txt Note This combines the HDFS files locally. Make sure you have enough disk space before running
Simple Solution: Download it from here and add to $HADOOP_HOME/bin (Source) IMPORTANT UPDATE: For hadoop-2.6.0 you can download binaries from Titus Barik blog >>. I have not only needed to point HADOOP_HOME to extracted directory [path], but also provide system property -Djava.library.path=[path]\bin to load native libs (dll).
Interesting question, I spent some time looking at the code for the details and here are my thoughts. The splits are handled by the client by InputFormat.getSplits, so a look at FileInputFormat gives the following info: For each input file, get the file length, the block size and calculate the split size as max(minSize, min(maxSize, … Read more
As mentioned by PinoSan this question is probably too generic, and you should be able to find your answer in any Spark Getting Started, or Tutorial. Let me point you to some interesting content: Spark Quick Start Guide Getting Started with Apache Spark, ebook Introduction to Apache Spark with Examples and Use Cases Disclaimer: I … Read more
I find the tutorials from hortonworks a rather good starting point http://hortonworks.com/tutorials/#tuts-developers To deep dive a must read is Tom White’s ‘Hadoop: The Definitve Guide’. ‘Hadoop in Practice’ shows a lot of cookbook like examples.