In order to keep everything on the grid use hadoop streaming with a single reducer and cat as the mapper and reducer (basically a noop) – add compression using MR flags. hadoop jar \ $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar \<br> -Dmapred.reduce.tasks=1 \ -Dmapred.job.queue.name=$QUEUE \ -input “$INPUT” \ -output “$OUTPUT” \ -mapper cat \ -reducer cat If you want compression … Read more
Check out this post from Alan Gates, Pig architect at Yahoo!, that compares when would use a SQL like Hive rather than Pig. He makes a very convincing case as to the usefulness of a procedural language like Pig (vs. declarative SQL) and its utility to dataflow designers.
As zjffdu said, the path expansion is done by the shell. One common way to solve your problem is to simply use Pig parameters (which is a good way to make your script more resuable anyway): shell: pig -f script.pig -param input=/user/training/test/{20100810..20100812} script.pig: temp = LOAD ‘$input’ USING SomeLoader() AS (…);
You can do it in 2 ways: 1. Write a UDF which returns a bag of tuples. It will be the most flexible solution, but requires Java code; 2. Write a rigid script like this: inpt = load ‘/pig_fun/input/pivot.txt’ as (Id, Column1, Column2, Column3); bagged = foreach inpt generate Id, TOBAG(TOTUPLE(‘Column1’, Column1), TOTUPLE(‘Column2’, Column2), TOTUPLE(‘Column3’, … Read more
Are you running a pig 0.12.0 or earlier jar against hadoop 2.2, if this is the case then I managed to get around this error by recompiling the pig jar from src, here is a summary of the steps involved on a debian type box download the pig-0.12.0.tar.gz unpack the jar and set permissions then … Read more
The message “ERROR 1066: Unable to open iterator for alias myAlias” suggests that there is something going wrong in the line where you use myAlias. However, usually you will see this error if something went wrong BEFORE you are trying to use this alias. So the first thing to do is look up further along … Read more