Modify and replay MongoDB oplog

by

One of the big issues in application or human error data corruption is that the offending write to the primary will immediately be replicated to the secondary.

This is one of the reasons that users take advantage of “slaveDelay” – an option to run one of your secondary nodes with a fixed time delay (of course that only helps you if you discover the error or bug during the time period that’s shorter than the delay on that secondary).

In case you don’t have such a set-up, you have to rely on a backup to recreate the state of the records you need to restore to their pre-bug state.

Perform all the operations on a separate stand-alone copy of your data – only after verifying that everything was properly recreated should you then move the corrected data into your production system.

What is required to be able to do this is a recent copy of the backup (let’s say the backup is X hours old) and the oplog on your cluster must hold more than X hours worth of data. I didn’t specify which node’s oplog because (a) every member of the replica set has the same contents in the oplog and (b) it is possible that your oplog size is different on different node members, in which case you want to check the “largest” one.

So let’s say your most recent backup is 52 hours old, but luckily you have an oplog that holds 75 hours worth of data (yay).

You already realized that all of your nodes (primary and secondaries) have the “bad” data, so what you would do is restore this most recent backup into a new mongod. This is where you will restore these records to what they were right before the offending update – and then you can just move them into the current primary from where they will get replicated to all the secondaries.

While restoring your backup, create a mongodump of your oplog collection via this command:

mongodump -d local -c oplog.rs -o oplogD

Move the oplog to its own directory renaming it to oplog.bson:

mkdir oplogR
mv oplogD/local/oplog.rs.bson oplogR/oplog.bson

Now you need to find the “offending” operation. You can dump out the oplog to human readable form, using the bsondump command on oplogR/oplog.bson file (and then use grep or what-not to find the “bad” update). Alternatively you can query against the original oplog in the replica set via use local and db.oplog.rs.find() commands in the shell.

Your goal is to find this entry and note its ts field.

It might look like this:

"ts" : Timestamp( 1361497305, 2789 )

Note that the mongorestore command has two options, one called --oplogReplay and the other called oplogLimit. You will now replay this oplog on the restored stand-alone server BUT you will stop before this offending update operation.

The command would be (host and port are where your newly restored backup is):

mongorestore -h host --port NNNN --oplogReplay --oplogLimit 1361497305:2789 oplogR

This will restore each operation from the oplog.bson file in oplogR directory stopping right before the entry with ts value Timestamp(1361497305, 2789).

Recall that the reason you were doing this on a separate instance is so you can verify the restore and replay created correct data – once you have verified it then you can write the restored records to the appropriate place in the real primary (and allow replication propagate the corrected records to the secondaries).

More Related Contents:

  1. MEAN app vs. Mongo service local in Angular2
  2. Find objects between two dates MongoDB
  3. MongoDB: Combine data from multiple collections into one..how?
  4. MongoDB: Is it possible to make a case-insensitive query?
  5. How to update the _id of one MongoDB Document?
  6. MongoDB: count the number of items in an array
  7. Multiple join conditions using the $lookup operator
  8. Aggregate $lookup Total size of documents in matching pipeline exceeds maximum document size
  9. MongoDB – admin user not authorized
  10. How can I run MongoDB as a Windows service?
  11. MongoDB query with an ‘or’ condition
  12. Mongo complex sorting?
  13. MongoDB: update every document on one field
  14. How to sum every fields in a sub document of MongoDB?
  15. Where does mongodb stand in the CAP theorem?
  16. MongoDb shuts down with Code 100 [closed]
  17. Mongodb Query To select records having a given key
  18. MongoDB: how to parse date in 3.6 mongoDb version?
  19. MongoDB – is DBREF necessary?
  20. saving picture to mongodb
  21. MongoDB – Querying between a time range of hours
  22. Delete a key from a MongoDB document using Mongoose
  23. Simple HTTP/TCP health check for MongoDB
  24. Mongoose.js instance.save() callback not firing
  25. MongoDB Update Deep Array
  26. Is there a convention to name collection in MongoDB?
  27. SocketException: Address already in use MONGODB
  28. Check if MongoDB upsert did an insert or an update
  29. Map-Reduce performance in MongoDb 2.2, 2.4, and 2.6
  30. Mongo $in operator performance

Leave a Comment