MongoDB select count(distinct x) on an indexed column – count unique results for large data sets

by

1) The easiest way to do this is via the aggregation framework. This takes two “$group” commands: the first one groups by distinct values, the second one counts all of the distinct values

pipeline = [ 
    { $group: { _id: "$myIndexedNonUniqueField"}  },
    { $group: { _id: 1, count: { $sum: 1 } } }
];

//
// Run the aggregation command
//
R = db.runCommand( 
    {
    "aggregate": "myCollection" , 
    "pipeline": pipeline
    }
);
printjson(R);

2) If you want to do this with Map/Reduce you can. This is also a two-phase process: in the first phase we build a new collection with a list of every distinct value for the key. In the second we do a count() on the new collection.

var SOURCE = db.myCollection;
var DEST = db.distinct
DEST.drop();


map = function() {
  emit( this.myIndexedNonUniqueField , {count: 1});
}

reduce = function(key, values) {
  var count = 0;

  values.forEach(function(v) {
    count += v['count'];        // count each distinct value for lagniappe
  });

  return {count: count};
};

//
// run map/reduce
//
res = SOURCE.mapReduce( map, reduce, 
    { out: 'distinct', 
     verbose: true
    }
    );

print( "distinct count= " + res.counts.output );
print( "distinct count=", DEST.count() );

Note that you cannot return the result of the map/reduce inline, because that will potentially overrun the 16MB document size limit. You can save the calculation in a collection and then count() the size of the collection, or you can get the number of results from the return value of mapReduce().

More Related Contents:

  1. MEAN app vs. Mongo service local in Angular2
  2. Find objects between two dates MongoDB
  3. MongoDB: Combine data from multiple collections into one..how?
  4. MongoDB: Is it possible to make a case-insensitive query?
  5. How to update the _id of one MongoDB Document?
  6. MongoDB: count the number of items in an array
  7. Multiple join conditions using the $lookup operator
  8. Aggregate $lookup Total size of documents in matching pipeline exceeds maximum document size
  9. MongoDB – admin user not authorized
  10. How can I run MongoDB as a Windows service?
  11. MongoDB query with an ‘or’ condition
  12. Mongo complex sorting?
  13. MongoDB: update every document on one field
  14. How to sum every fields in a sub document of MongoDB?
  15. Where does mongodb stand in the CAP theorem?
  16. MongoDb shuts down with Code 100 [closed]
  17. Mongodb Query To select records having a given key
  18. MongoDB 3.2 authentication failed
  19. MongoDB ‘unable to find index for $geoNear query’
  20. MongoDB sort documents by array elements
  21. Does Mongoose support the Mongodb `findAndModify` method?
  22. MongoDB Text Search AND multiple search words
  23. How to use dot in field name?
  24. What is the maximum number of parameters passed to $in query in MongoDB?
  25. Mongodb aggregation pipeline how to limit a group push
  26. MongoDB : aggregation framework : $match between fields
  27. Modify and replay MongoDB oplog
  28. Creating custom Object ID in MongoDB
  29. Sort nested array of objects
  30. How to create a Mongo Docker Image with default collections and data?

Leave a Comment