Map-Reduce performance in MongoDb 2.2, 2.4, and 2.6

by

Here is the current state of functionality for Map/Reduce in MongoDB

1) Most of the performance limitations for Map/Reduce still remain in MongoDB version 2.2. The Map/Reduce engine still requires that every record get converted from BSON to JSON, the actual calculations are performed using the embedded JavaScript engine (which is slow), and there still is a single global JavaScript lock, which only allows a single JavaScript thread to run at a single time.

There have been some incremental improvements to Map/Reduce for sharded clusters. Most notably, the final Reduce operation is now distributed across multiple shards, and the output is also sharded in parallel.

I would not recommend Map/Reduce for real-time aggregation in MongoDB version 2.2

2) Starting with MongoDB 2.2, there is now a new Aggregation Framework. This is a new implementation of aggregation operations, written in C++, and tightly integrated into the MongoDB framework.

Most Map/Reduce jobs can be rewritten to use the Aggregation Framework. They usually run faster (20x speed improvement vs. Map/Reduce is common in version 2.2), they make full use of the existing query engine, and you can run multiple Aggregation commands in parallel.

If you have real-time aggregation requirements, the first place to start is with the Aggregation Framework. For more information about the aggregation framework, take a look at these links:

3) There have been significant improvements in Map/Reduce in MongoDB version 2.4. The SpiderMonkey JavaScript engine has been replaced by the V8 JavaScript engine, and there is no longer a global JavaScript lock, which means that multiple Map/Reduce threads can run concurrently.

The Map/Reduce engine is still considerably slower than the aggregation framework, for two main reasons:

  • The JavaScript engine is interpreted, while the Aggregation Framework
    runs compiled C++ code

  • The JavaScript engine still requires that every document being examined get converted from BSON to JSON; if you’re saving the output in a collection, the result set must then be converted from JSON back to BSON

There are no significant changes in Map/Reduce between 2.4 and 2.6.

I still do not recommend using the Map/Reduce for real-time aggregation in MongoDB version 2.4 or 2.6.

4) If you really need Map/Reduce, you can also look at the Hadoop Adaptor. There’s more information here:

More Related Contents:

  1. Does MongoDB’s $in clause guarantee order
  2. Check if every element in array matches condition
  3. Using map/reduce for mapping the properties in a collection
  4. Merging two collections in MongoDB
  5. MongoDB aggregation comparison: group(), $group and MapReduce
  6. Find all duplicate documents in a MongoDB collection by a key field
  7. Remove Duplicates from MongoDB
  8. Group result by 15 minutes time interval in MongoDb
  9. MongoDB Query Help – query on values of any key in a sub-object
  10. How to join multiple collections with $lookup in mongodb
  11. How can I rename a field for all documents in MongoDB?
  12. Mongoose Unique index not working!
  13. Reducing MongoDB database file size
  14. Checking if a field contains a string
  15. MongoDB : find value in Array with multiple criteria
  16. mongodb count num of distinct values per field/key
  17. String field value length in mongoDB
  18. How do you rename a MongoDB database?
  19. mongodb how to get max value from collections
  20. How to aggregate by year-month-day on a different timezone
  21. Problems to run examples in Meteor
  22. How to resolve MongoDB failing to instantiate [closed]
  23. How to use Elasticsearch with MongoDB?
  24. What is a Cursor in MongoDB?
  25. Meteor: difference between names for collections, variables, publications, and subscriptions?
  26. MongoDB – how to query for a nested item inside a collection?
  27. What’s the fastest way to copy a collection within the same database?
  28. Mongoose: deep population (populate a populated field)
  29. How do I rename fields when performing search/projection in MongoDB?
  30. mongodb aggregation framework match by nested documents

Leave a Comment