I was able to reproduce your problem. The problem is where you are creating the jar. Basically, the directory that you are packaging into the jar is confusing the jar file in locating the main class file. Instead if you try doing : /usr/lib/jvm/jdk1.7.0_07/bin/jar cf Dictionary.jar /home/hduser/dir/Dictionary.class i.e. package the class file specifically into the … Read more
How Many Reduces? ( From official documentation) The right number of reduces seems to be 0.95 or 1.75 multiplied by (no. of nodes) * (no. of maximum containers per node). With 0.95 all of the reduces can launch immediately and start transferring map outputs as the maps finish. With 1.75 the faster nodes will finish … Read more
The answer by @user1668782 is a great explanation for the question and I’ll try to give a graphical depiction of it. Assume we have a file of 400MB with consists of 4 records(e.g : csv file of 400MB and it has 4 rows, 100MB each) If the HDFS Block Size is configured as 128MB, then … Read more
In support of the chosen answer I add: Following on from this explanation **Input**: symbol time price a 1 10 a 2 20 b 3 30 **Map output**: create composite key\values like so: > symbol-time time-price > >**a-1** 1-10 > >**a-2** 2-20 > >**b-3** 3-30 The Partitioner: will route the a-1 and a-2 keys to the same reducer … Read more
I/O operations often come with significant overhead that will occur on the master node. Since this work isn’t parallelized, it can take quite a bit of time. And since it is not a job, it does not show up in the resource manager UI. Some examples of I/O tasks that are done by the master … Read more
If you are using hadoop 2.* API there are more elegant solutions: Configuration conf = getConf(); Job job = Job.getInstance(conf); FileSystem fs = FileSystem.get(conf); //the second boolean parameter here sets the recursion to true RemoteIterator<LocatedFileStatus> fileStatusListIterator = fs.listFiles( new Path(“path/to/lib”), true); while(fileStatusListIterator.hasNext()){ LocatedFileStatus fileStatus = fileStatusListIterator.next(); //do stuff with the file like … job.addFileToClassPath(fileStatus.getPath()); }
As zjffdu said, the path expansion is done by the shell. One common way to solve your problem is to simply use Pig parameters (which is a good way to make your script more resuable anyway): shell: pig -f script.pig -param input=/user/training/test/{20100810..20100812} script.pig: temp = LOAD ‘$input’ USING SomeLoader() AS (…);
The parameter mapred.max.split.size which can be set per job individually is what you looking for. Don’t change dfs.block.size because this is global for HDFS and can lead to problems.
This error is caused by the block replication system of HDFS since it could not manage to make any copies of a specific block within the focused file. Common reasons of that: Only a NameNode instance is running and it’s not in safe-mode There is no DataNode instances up and running, or some are dead. … Read more
LOAD DATA just copies the files to hive datafiles. Hive does not do any transformation while loading data into tables. So, in this case the input file /home/user/test_details.txt needs to be in ORC format if you are loading it into an ORC table. A possible workaround is to create a temporary table with STORED AS … Read more