What does it mean to fit “working set” into RAM for MongoDB?

by

“Working set” is basically the amount of data AND indexes that will be active/in use by your system.

So for example, suppose you have 1 year’s worth of data. For simplicity, each month relates to 1GB of data giving 12GB in total, and to cover each month’s worth of data you have 1GB worth of indexes again totalling 12GB for the year.

If you are always accessing the last 12 month’s worth of data, then your working set is: 12GB (data) + 12GB (indexes) = 24GB.

However, if you actually only access the last 3 month’s worth of data, then your working set is: 3GB (data) + 3GB (indexes) = 6GB. In this scenario, if you had 8GB RAM and then you started regularly accessing the past 6 month’s worth of data, then your working set would start to exceed past your available RAM and have a performance impact.

But generally, if you have enough RAM to cover the amount of data/indexes you expect to be frequently accessing then you will be fine.

Edit: Response to question in comments
I’m not sure I quite follow, but I’ll have a go at answering. Firstly, the calculation for working set is a “ball park figure”. Secondly, if you have a (e.g.) 1GB index on user_id, then only the portion of that index that is commonly accessed needs to be in RAM (e.g. suppose 50% of users are inactive, then 0.5GB of the index will be more frequently required/needed in RAM). In general, the more RAM you have, the better especially as working set is likely to grow over time due to increased usage. This is where sharding comes in – split the data over multiple nodes and you can cost effectively scale out. Your working set is then divided over multiple machines, meaning the more can be kept in RAM. Need more RAM? Add another machine to shard on to.

More Related Contents:

  1. Find in Double Nested Array MongoDB
  2. MongoDB: upsert sub-document
  3. Find MongoDB records where array field is not empty
  4. How do I update Array Elements matching criteria in a MongoDB document?
  5. MongoDB – Update an object in nested Array
  6. Get a count of total documents with MongoDB when using limit
  7. Creating Multifield Indexes in Mongoose / MongoDB
  8. To what level does MongoDB lock on writes? (or: what does it mean by “per connection”
  9. Mongo order by length of array
  10. Mongodb Explain for Aggregation framework
  11. MongoDB aggregation comparison: group(), $group and MapReduce
  12. Exception: can’t convert from BSON type EOO to Date
  13. Multiple Counts with single query in mongodb
  14. Failed to start mongod.service: Unit mongod.service not found
  15. Mongodb Aggregation Framework: Does $group use index?
  16. Using MongoDB $pull to delete documents within an Array
  17. return query based on date
  18. MongoParseError: options useCreateIndex, useFindAndModify are not supported
  19. What is a Cursor in MongoDB?
  20. How do I rename fields when performing search/projection in MongoDB?
  21. Update MongoDB collection using $toLower
  22. mongodb aggregation framework match by nested documents
  23. How to check if an array field contains a unique value or another array in MongoDB?
  24. MongoDB.service failed with result exit-code
  25. How can I sort a Meteor collection by time of insertion?
  26. MongoDB index/RAM relationship
  27. How to do pagination using range queries in MongoDB?
  28. Export mongodb aggregation framework result to a new collection
  29. MongoDB aggregate by field exists
  30. MongoDB {aggregation $match} vs {find} speed

Leave a Comment