When you iterate over a list, the loop variable receives the actual list elements, not their indices. Thus, in your example s is a string (first abc, then def). It looks like what you’re trying to do is essentially this: orig_list = [‘abc’, ‘def’] map_list = [(el, 1) for el in orig_list] This is using … Read more
Functionality wise there is not much difference between the old (o.a.h.mapred) and the new (o.a.h.mapreduce) API. The only significant difference is that records are pushed to the mapper/reducer in the old API. While the new API supports both pull/push mechanism. You can get more information about the pull mechanism here. Also, the old API has … Read more
This is similar to a question that was asked on the MongoDB-users Google Groups. https://groups.google.com/group/mongodb-user/browse_thread/thread/60a8b683e2626ada?pli=1 The answer references an on-line tutorial which looks similar to your example: http://tebros.com/2011/07/using-mongodb-mapreduce-to-join-2-collections/ For more information on MapReduce in MongoDB, please see the documentation: http://www.mongodb.org/display/DOCS/MapReduce Additionally, there is a useful step-by-step walkthrough of how a MapReduce operation works in the … Read more
Every test I have personally run (including using your own data) shows aggregation framework being a multiple faster than map reduce, and usually being an order of magnitude faster. Just taking 1/10th of the data you posted (but rather than clearing OS cache, warming the cache first – because I want to measure performance of … Read more
OK, this is a little more complex because you’ll need to use some recursion. To make the recursion happen, you’ll need to be able to store some functions on the server. Step 1: define some functions and put them server-side isArray = function (v) { return v && typeof v === ‘object’ && typeof v.length … Read more
First of all shuffling is the process of transfering data from the mappers to the reducers, so I think it is obvious that it is necessary for the reducers, since otherwise, they wouldn’t be able to have any input (or input from every mapper). Shuffling can start even before the map phase has finished, to … Read more
Oh please don’t do it with runJar, the Java API is very good. See how you can start a job from normal code: // create a configuration Configuration conf = new Configuration(); // create a new job based on the configuration Job job = new Job(conf); // here you have to put your mapper class … Read more
The query you want is this: db.collection.find({“users”:{“$not”:{“$elemMatch”:{“user”:{$nin:[1,5,7]}}}}}) This says find me all documents that don’t have elements that are outside of the list 1,5,7.
The number of map tasks for a given job is driven by the number of input splits and not by the mapred.map.tasks parameter. For each input split a map task is spawned. So, over the lifetime of a mapreduce job the number of map tasks is equal to the number of input splits. mapred.map.tasks is … Read more
You should also properly configure the maximum memory allocations for MapReduce. From this HortonWorks tutorial: […] Each machine in our cluster has 48 GB of RAM. Some of this RAM should be >reserved for Operating System usage. On each node, we’ll assign 40 GB RAM for >YARN to use and keep 8 GB for the … Read more