How to remove duplicates based on a key in Mongodb?

by

This answer is obsolete : the dropDups option was removed in MongoDB 3.0, so a different approach will be required in most cases. For example, you could use aggregation as suggested on: MongoDB duplicate documents even after adding unique key.

If you are certain that the source_references.key identifies duplicate records, you can ensure a unique index with the dropDups:true index creation option in MongoDB 2.6 or older:

db.things.ensureIndex({'source_references.key' : 1}, {unique : true, dropDups : true})

This will keep the first unique document for each source_references.key value, and drop any subsequent documents that would otherwise cause a duplicate key violation.

Important Note: Any documents missing the source_references.key field will be considered as having a null value, so subsequent documents missing the key field will be deleted. You can add the sparse:true index creation option so the index only applies to documents with a source_references.key field.

Obvious caution: Take a backup of your database, and try this in a staging environment first if you are concerned about unintended data loss.

More Related Contents:

  1. Fastest way to remove duplicate documents in mongodb
  2. Find all duplicate documents in a MongoDB collection by a key field
  3. Toggle a boolean value with mongoDB
  4. How to remove array element in mongodb?
  5. MongoDB: How to update multiple documents with a single command?
  6. Can you specify a key for $addToSet in Mongo?
  7. How to copy a collection from one database to another in MongoDB
  8. What is the default database path for MongoDB?
  9. How to aggregate sum in MongoDB to get a total count?
  10. mongo – couldn’t connect to server 127.0.0.1:27017
  11. What is the best way to store dates in MongoDB?
  12. How to work around the lack of transactions in MongoDB?
  13. What’s the $unwind operator in MongoDB?
  14. Order of responses to MongoDB $in query? [duplicate]
  15. How do I copy a database from one MongoDB server to another?
  16. mongoDB prefix wildcard: fulltext-search ($text) find part with search-string
  17. Pull and addtoset at the same time with mongo
  18. Is there a convention to name collection in MongoDB?
  19. SocketException: Address already in use MONGODB
  20. Check if MongoDB upsert did an insert or an update
  21. Limiting results in MongoDB but still getting the full count?
  22. Mongodb : $in operator vs lot of single queries
  23. What does “too many positional options” mean when doing a mongoexport?
  24. MongoDB: Only fetch documents created in the last 24hrs?
  25. $project in $lookup mongodb
  26. How to use 3rd party libraries in glassfish?
  27. mongodb service is not starting up
  28. Is there a simple way to export the data from a meteor deployed app?
  29. Using findOne in mongodb to get element with max id
  30. Mongo $in operator performance

Leave a Comment