hadoop-partitioning - w3toppers.com

What is the use of grouping comparator in hadoop map reduce

In support of the chosen answer I add: Following on from this explanation **Input**: symbol time price a 1 10 a 2 20 b 3 30 **Map output**: create composite key\values like so: > symbol-time time-price > >**a-1**         1-10 > >**a-2**         2-20 > >**b-3**         3-30 The Partitioner: will route the a-1 and a-2 keys to the same reducer … Read more

hadoop map reduce secondary sorting

I find it easy to understand certain concepts with help of diagrams and this is certainly one of them. Lets assume that our secondary sorting is on a composite key made out of Last Name and First Name. With the composite key out of the way, now lets look at the secondary sorting mechanism The … Read more

Specify minimum number of generated files from Hive insert

The number of files generated during INSERT … SELECT depends on the number of processes running on final reducer (final reducer vertex if you are running on Tez) plus bytes per reducer configured. If the table is partitioned and there is no DISTRIBUTE BY specified, then in the worst case each reducer creates files in … Read more

Sqoop import : composite primary key and textual primary key

Specify split column manually. Split column is not necessarily equal to PK. You can have complex PK and some int Split column. You can specify any integer column or even simple function (some simple function like substring or cast, not aggregation or analytic). Split column preferably should be evenly distributed integer. For example if your … Read more