Database NotesRohit’s interview study guide
Chapter 22

Transactions, Replication and Sharding

How MongoDB keeps a write safe: atomic single documents, multi-document transactions, replica sets with the oplog, write and read concern, and sharding for scale, all run on real three-member and sharded clusters.

9 topics
Parts marked Advanced are extra depth. Skip them on a first read or a quick revision.

#Atomicity at the document level

Every write to a single document is atomic in MongoDB. One updateOne can change ten fields, push into an array and edit an embedded sub-document, and other clients see either the document before the whole update or after it, never half of it. This has been true since the first version, long before multi-document transactions existed, and it's why MongoDB's schema advice says "embed what you change together": if an order's items, totals and status live in one document, a single update keeps them consistent with no transaction at all.

A useful mental model: a document is a sealed envelope. The server opens it, makes all your changes, and seals it again before anyone else can look. Two envelopes are two separate seals, though: an updateMany that touches 50 documents makes 50 separate atomic writes, and a failure on the 30th leaves the first 29 changed.

The main example: add an item to order 2 and keep its itemCount, total and updatedAt in step, in one statement. The data is the site's sample shop (customers, products, orders with embedded items) loaded into the ch22 database on a three-member replica set.

mongosh
db.orders.updateOne(
  { _id: 2 },
  {
    $push: { items: { productId: 7, qty: 1, price: 29 } },
    $inc: { itemCount: 1, total: 29 },
    $set: { updatedAt: new Date("2026-10-16T12:00:00Z") }
  }
)
db.orders.findOne({ _id: 2 })
Result
{
  acknowledged: true,
  insertedId: null,
  matchedCount: 1,
  modifiedCount: 1,
  upsertedCount: 0
}
{
  _id: 2,
  customerId: 1,
  status: 'paid',
  itemCount: 3,
  total: 117,
  items: [
    { productId: 5, qty: 1, price: 39 },
    { productId: 6, qty: 1, price: 49 },
    { productId: 7, qty: 1, price: 29 }
  ],
  updatedAt: ISODate('2026-10-16T12:00:00.000Z')
}
What’s happening
  1. The filter { _id: 2 } finds exactly one document: Rohit's paid order with two books, itemCount: 2 and total: 88.
  2. Three update operators run against that one document: $push appends the charger to items, $inc adds 1 to itemCount (2 → 3) and 29 to total (88 → 117), and $set writes updatedAt.
  3. The server applies all three while holding the document's write intent, then commits the new version as one unit. A reader running at the same moment gets either the old document (2 items, 88) or the new one (3 items, 117), never "3 items but total still 88".
  4. matchedCount: 1, modifiedCount: 1 confirms one document changed. If the server crashed half-way, recovery would bring back the old version, not a mix.
  5. This is the whole idea: when related values live in one document, one ordinary update keeps them consistent. No session, no transaction, no extra round trips.
AdvancedEdge case: updateMany is not all-or-nothing

updateMany is many single-document writes in a row. If one fails, the ones before it stay changed. Here a unique index on email makes the second Toronto customer's update fail:

mongosh
db.customers.createIndex({ email: 1 }, { unique: true })
db.customers.updateMany({ city: "Toronto" }, { $set: { email: "toronto@example.com" } })
db.customers.find({ city: "Toronto" }, { name: 1, email: 1 })
Result
email_1
Uncaught MongoServerError: Plan executor error during update :: caused by :: E11000 duplicate key error collection: ch22.customers index: email_1 dup key: { email: "toronto@example.com" }
[
  { _id: 1, name: 'Rohit', email: 'toronto@example.com' },
  { _id: 5, name: 'Noah', email: 'noah@example.com' }
]
What’s happening
  1. Two customers match city: "Toronto": Rohit (_id: 1) and Noah (_id: 5).
  2. The first document is updated and committed: Rohit's email becomes toronto@example.com.
  3. The second write would give Noah the same email, which the unique index rejects with E11000, and the command stops with an error.
  4. Rohit's change is not undone. The command failed, yet the collection is half-updated. In SQL an UPDATE statement is atomic as a whole; in MongoDB only each document is.
  5. If you need "all of them or none", wrap the updateMany in a transaction (next topic) or redesign so the data lives in one document.
AdvancedPitfall: the read-then-write race

The most common concurrency bug in MongoDB code isn't about transactions at all: the app reads a document, decides in code, then writes. Between the read and the write, other clients do the same. Eight shells try to buy the last five MacBook Airs at the same moment, first the broken way:

buy-race.js
// The broken version: read, decide in the app, then write
const p = db.products.findOne({ _id: 2 });
sleep(200);                                   // the app "thinks" (any real work between read and write)
if (p.stock >= 1) {
  db.products.updateOne({ _id: 2 }, { $set: { stock: p.stock - 1 } });
  print("sold, stock was " + p.stock);
} else print("sold out");
Result
shell 2: sold, stock was 5
shell 6: sold, stock was 5
shell 3: sold, stock was 5
shell 5: sold, stock was 5
shell 1: sold, stock was 5
shell 4: sold, stock was 5
shell 8: sold, stock was 5
shell 7: sold, stock was 5
{ _id: 2, name: 'MacBook Air', stock: 4 }
What’s happening
  1. All eight shells run findOne within a few milliseconds of each other, so all eight read stock: 5.
  2. Each one decides "5 ≥ 1, I can sell" and writes $set: { stock: 4 }, a value computed from its own stale read.
  3. Every write is atomic, but each one simply overwrites the last: eight sales were promised and the stock went 5 → 4. That's a lost update.
  4. The single-document atomicity guarantee didn't help because the decision happened outside the database.
AdvancedFix: the check inside the filter

The fix is to put the condition inside the update, so the check and the change are one atomic operation:

buy.js
// "Buy one MacBook Air", run by 8 shells at the same moment
const r = db.products.updateOne(
  { _id: 2, stock: { $gte: 1 } },   // only if there is one left
  { $inc: { stock: -1 } }
);
print(r.modifiedCount === 1 ? "sold" : "sold out");
Result
shell 3: sold
shell 1: sold
shell 4: sold
shell 6: sold
shell 5: sold out
shell 2: sold
shell 7: sold out
shell 8: sold out
{ _id: 2, name: 'MacBook Air', stock: 0 }
What’s happening
  1. Each shell sends one updateOne whose filter says "product 2 and stock is at least 1".
  2. The server evaluates the filter and applies $inc: -1 atomically per document. Two updates to the same document are serialised, so the fifth sale sees stock: 1 and takes it to 0.
  3. The sixth, seventh and eighth updates find no document matching stock: { $gte: 1 }, so modifiedCount is 0 and they print "sold out".
  4. Exactly five sales, stock exactly 0. This compare-and-set pattern (filter on the expected state, change it atomically) is how most MongoDB apps avoid transactions altogether. It's the same idea as optimistic locking in SQL, see Optimistic vs pessimistic locking.
AdvancedfindOneAndUpdate returns the document it changed

When you need the result of the change (the new stock, the next counter value, the claimed job), findOneAndUpdate does the match, the update and the read back as one atomic step:

mongosh
db.products.findOneAndUpdate(
  { _id: 3, stock: { $gte: 2 } },
  { $inc: { stock: -2 } },
  { returnDocument: "after", projection: { name: 1, stock: 1 } }
)
db.products.findOneAndUpdate(
  { _id: 4, stock: { $gte: 1 } },
  { $inc: { stock: -1 } },
  { returnDocument: "after", projection: { name: 1, stock: 1 } }
)
Result
{ _id: 3, name: 'Pixel 10', stock: 18 }
null
What’s happening
  1. The Pixel 10 has stock: 20, so the first call matches, decrements by 2 and, because of returnDocument: "after", returns the document as it is after the update: 18.
  2. The iPhone 17 has stock: 0, so the filter stock: { $gte: 1 } matches nothing and the call returns null. Nothing was changed.
  3. No other client can slip in between the match and the returned value, which makes this the building block for job queues ("claim the next pending job") and counters.
  4. With returnDocument: "before" (the default) you'd get the old version instead.
Predict the stock

Product 2 has stock: 5. Eight clients run this at the same time. What's the final stock, and how many clients think they sold one?

mongosh
const p = db.products.findOne({ _id: 2 });
sleep(200);   // the app does some work
if (p.stock > 0) db.products.updateOne({ _id: 2 }, { $inc: { stock: -1 } });
Show answer

The final stock is -3, and all eight think they sold one.

What’s happening
  1. All eight findOne calls run before any update lands, so every client reads stock: 5 and passes the p.stock > 0 check.
  2. This time the write is $inc: -1, not $set: p.stock - 1, so there's no lost update: each decrement is applied atomically to the current value.
  3. Eight decrements of 5 gives 5 → -3. The $inc fixed the lost-update bug but not the oversell, because the check still happened in the app on a stale value.
  4. The fix is to move the check into the filter: updateOne({ _id: 2, stock: { $gt: 0 } }, { $inc: { stock: -1 } }), which stops at exactly 0.

#Multi-document transactions

Sometimes one business action really does change several documents in different collections: place an order and reduce stock, move money from account A and to account B. For those, MongoDB has multi-document transactions (replica sets since 4.0, sharded clusters since 4.2). They behave like SQL transactions: all the writes commit together or none do, other clients don't see them until commit, and reads inside the transaction see one consistent snapshot.

Transactions live on a session. You start a session, start a transaction on it, pass the session to every operation that belongs to the transaction, then commitTransaction() or abortTransaction(). An operation that doesn't carry the session is outside the transaction, even if it runs in the middle of it. Think of the session as a shopping basket: things only go through the till together if you put them in the basket.

From my notes: "Example of a transaction using SESSION & COMMIT & ABORT". Here it is in mongosh, placing order 4 for two Pixel 10s.

mongosh
const session = db.getMongo().startSession();
const shop = session.getDatabase("ch22");
session.startTransaction({ readConcern: { level: "snapshot" }, writeConcern: { w: "majority" } });

shop.products.updateOne({ _id: 3, stock: { $gte: 2 } }, { $inc: { stock: -2 } })
shop.orders.insertOne({ _id: 4, customerId: 3, status: "paid", itemCount: 1, total: 1598,
                        items: [{ productId: 3, qty: 2, price: 799 }] })
db.orders.findOne({ _id: 4 })          // outside the transaction: not visible yet

session.commitTransaction()
session.endSession()

db.orders.findOne({ _id: 4 }, { status: 1, total: 1 })
db.products.findOne({ _id: 3 }, { name: 1, stock: 1 })
Result
{
  acknowledged: true,
  insertedId: null,
  matchedCount: 1,
  modifiedCount: 1,
  upsertedCount: 0
}
{ acknowledged: true, insertedId: 4 }
null
{ _id: 4, status: 'paid', total: 1598 }
{ _id: 3, name: 'Pixel 10', stock: 18 }
What’s happening
  1. startSession() creates a logical session (a server-side id). session.getDatabase("ch22") returns a database handle bound to that session, so every operation through shop belongs to it.
  2. startTransaction opens the transaction with snapshot reads and a majority write concern for the commit. Nothing is sent to the server yet; the transaction really starts with the first operation.
  3. The stock update (20 → 18 inside the transaction) and the order insert run, and both report success, but they're only visible inside the transaction.
  4. db.orders.findOne({ _id: 4 }) uses the plain db, which isn't bound to the session, so it's an outside reader. It returns null: uncommitted writes are invisible to everyone else.
  5. commitTransaction() makes both writes visible at once and waits for a majority of the replica set to have them. After that, the order exists and the stock is 18 for every reader. Either both changes happened or, on any failure before commit, neither would have.
AdvancedAborting when a business rule fails

abortTransaction() throws away everything the transaction did. Here order 5 wants six MacBook Airs but only five are in stock, so the app aborts after the conditional update matches nothing:

mongosh
const session = db.getMongo().startSession();
const shop = session.getDatabase("ch22");
session.startTransaction();
shop.orders.insertOne({ _id: 5, customerId: 4, status: "paid", itemCount: 1, total: 7794,
                        items: [{ productId: 2, qty: 6, price: 1299 }] })
const r = shop.products.updateOne({ _id: 2, stock: { $gte: 6 } }, { $inc: { stock: -6 } })
if (r.modifiedCount === 0) { session.abortTransaction(); print("not enough stock: aborted"); }
else session.commitTransaction();
session.endSession()

db.orders.findOne({ _id: 5 })
db.products.findOne({ _id: 2 }, { name: 1, stock: 1 })
Result
{ acknowledged: true, insertedId: 5 }
not enough stock: aborted
null
{ _id: 2, name: 'MacBook Air', stock: 5 }
What’s happening
  1. The order insert succeeds inside the transaction (insertedId: 5), so at this point the transaction holds one write.
  2. The stock update's filter needs stock: { $gte: 6 }, but stock is 5, so modifiedCount is 0.
  3. The app sees that and calls abortTransaction(). The server discards the transaction's writes, including the order insert that had already "succeeded".
  4. Afterwards order 5 doesn't exist (null) and stock is still 5. That's atomicity across two collections: the insert and the update are one unit.
AdvancedwithTransaction retries for you

Drivers and mongosh have a withTransaction helper: it starts the transaction, runs your callback, commits, and retries the whole callback on TransientTransactionError and the commit on UnknownTransactionCommitResult. In application code this is the form to use (see Transactions from Node for the Node version).

mongosh
const session = db.getMongo().startSession();
session.withTransaction(() => {
  const shop = session.getDatabase("ch22");
  shop.products.updateOne({ _id: 8 }, { $inc: { stock: -1 } });
  shop.orders.insertOne({ _id: 6, customerId: 6, status: "paid", itemCount: 1, total: 249,
                          items: [{ productId: 8, qty: 1, price: 249 }] });
})
session.endSession()
db.products.findOne({ _id: 8 }, { stock: 1 })
Result
{ _id: 8, stock: 7 }
What’s happening
  1. withTransaction calls startTransaction, then your callback with the two writes, then commitTransaction. You never call commit yourself.
  2. If either write had hit a write conflict, the helper would abort and run the callback again from the top, which is why the callback must be safe to re-run (no side effects like sending an email inside it).
  3. The headphones' stock goes 8 → 7 and order 6 exists, committed together.
AdvancedTwo transactions writing the same document

MongoDB doesn't make the second writer wait for the first transaction. It fails fast with a write conflict, aborts that transaction, and labels the error so the app knows a retry is safe:

mongosh
const a = db.getMongo().startSession();
const b = db.getMongo().startSession();
a.startTransaction(); b.startTransaction();
a.getDatabase("ch22").products.updateOne({ _id: 1 }, { $inc: { stock: -1 } })
try { b.getDatabase("ch22").products.updateOne({ _id: 1 }, { $inc: { stock: -1 } }) }
catch (e) { print(e.codeName, e.errorLabels) }
a.commitTransaction()
try { b.commitTransaction() } catch (e) { print(e.codeName, e.errorLabels) }
db.products.findOne({ _id: 1 }, { stock: 1 })
Result
{
  acknowledged: true,
  insertedId: null,
  matchedCount: 1,
  modifiedCount: 1,
  upsertedCount: 0
}
WriteConflict [ 'TransientTransactionError' ]
NoSuchTransaction [ 'TransientTransactionError' ]
{ _id: 1, stock: 11 }
What’s happening
  1. Transaction a updates the ThinkBook (12 → 11 inside a) and now holds that document's write lock until it commits or aborts.
  2. Transaction b tries to update the same document. Instead of waiting, the server returns WriteConflict with the label TransientTransactionError, and b is aborted on the server.
  3. a commits normally. Trying to commit b now fails with NoSuchTransaction, also labelled transient, because b no longer exists.
  4. Final stock is 11: only a's decrement happened. The correct reaction to the label is to rerun b from the start, which withTransaction does automatically. Under heavy contention this retry loop is where transactions get expensive (next topic).
AdvancedThe 60-second limit and other server limits

These are the real defaults on MongoDB 9.0:

mongosh
db.adminCommand({ getParameter: 1, transactionLifetimeLimitSeconds: 1,
                  maxTransactionLockRequestTimeoutMillis: 1, maxConcurrentMultiDocumentTransactions: 1 })
Result
{
  maxConcurrentMultiDocumentTransactions: 10000,
  maxTransactionLockRequestTimeoutMillis: 5,
  transactionLifetimeLimitSeconds: 60,
  ...
}
AdvancedDemo: a transaction outliving its limit

A transaction that runs longer than transactionLifetimeLimitSeconds is aborted by a background job. To see it without waiting a minute, the limit is lowered to 3 seconds here:

mongosh
db.adminCommand({ setParameter: 1, transactionLifetimeLimitSeconds: 3 }).was
const s = db.getMongo().startSession();
s.startTransaction();
s.getDatabase("ch22").products.updateOne({ _id: 7 }, { $inc: { stock: -1 } }).modifiedCount
sleep(6000)   // the app is slow: an HTTP call, a big loop...
s.commitTransaction()
db.adminCommand({ setParameter: 1, transactionLifetimeLimitSeconds: 60 }).was
Result
60
1
Uncaught MongoServerError[NoSuchTransaction]: Transaction with { txnNumber: 1 } has been aborted.
3
What’s happening
  1. setParameter lowers the lifetime to 3 seconds; .was shows the old value, 60.
  2. The transaction's update succeeds (modifiedCount 1), then the app sleeps 6 seconds, like an app that makes a slow HTTP call while holding a transaction open.
  3. Meanwhile the server's periodic cleanup sees a transaction older than 3 seconds and aborts it, releasing its locks.
  4. The commit therefore fails with NoSuchTransaction: there's nothing left to commit. The last line restores the 60-second default (.was is now 3).
  5. The lesson: keep transactions short and never wait on anything external inside one. maxTransactionLockRequestTimeoutMillis: 5 means a transaction waits only 5 ms for a lock before aborting, and maxConcurrentMultiDocumentTransactions (new in 9.0) caps open transactions at 10,000, after which new ones fail with TooManyOpenTransactions.
AdvancedTransactions need a replica set

On a standalone mongod there's no transaction support at all. mongosh first trips over retryable writes, which also need a replica set; with retryWrites=false, the server's own message comes through:

mongosh
// on a standalone mongod started without --replSet, connected with ?retryWrites=false
db.runCommand({ insert: "products", documents: [{ _id: 1 }], lsid: { id: UUID() },
                txnNumber: Long(1), startTransaction: true, autocommit: false })
Result
MongoServerError: Transaction numbers are only allowed on a replica set member or mongos
What’s happening
  1. A transaction is identified by a session id (lsid) plus a transaction number (txnNumber); that's what the driver sends under the hood when you call startTransaction.
  2. A standalone has no oplog and no replication machinery to track transactions, so it rejects any command carrying a txnNumber.
  3. With mongosh's default retryWrites=true, the same attempt shows the driver's rewritten message instead: This MongoDB deployment does not support retryable writes. Please add retryWrites=false to your connection string.
  4. The fix for development is a single-node replica set: start mongod --replSet rs0 and run rs.initiate() once. That's exactly what this site's sandbox does.
AdvancedOlder versions: before 4.0

Before MongoDB 4.0 (2018) there were no multi-document transactions. Apps that needed them used the two-phase commit pattern: write a "transaction" document with state pending, apply each change while recording the transaction id on the documents, mark it applied, clean up, and write recovery code that rolls forward or back after a crash. You'll still find it in old codebases and old blog posts. Other history worth knowing:

  • 4.0: transactions on replica sets, limited to 16 MB of changes (one oplog entry) and to existing collections.
  • 4.2: distributed transactions on sharded clusters, and the 16 MB total limit removed (a transaction can now write several oplog entries; each entry is still at most 16 MB).
  • 4.4: creating collections and indexes inside a transaction.
  • 9.0: maxConcurrentMultiDocumentTransactions caps open transactions (default 10,000).

#When not to use transactions

Transactions work well, but they aren't free, and MongoDB's whole data model is built so you rarely need them. From my notes, three cases where you shouldn't reach for one: single-document operations (already atomic), high-frequency updates (the overhead becomes a bottleneck) and read-heavy workloads ("can lock resources and reduce concurrency… design around eventual consistency"). The first two are right. The third needs correcting, below.

The mental model: a transaction is a reservation. While it's open, it holds write locks on every document it changed and pins an old snapshot of the data in memory so its reads stay consistent. Short reservations are cheap. Many reservations competing for the same table, or reservations held for a long time, are where the cost appears.

The main example measures "high-frequency updates": eight workers each bump the same counter document 100 times, first with plain $inc, then with one transaction per bump (retrying on conflict, as withTransaction would).

hot-counter.js
// Worker A: 100 plain $inc updates of the same document
for (let i = 0; i < 100; i++) db.counters.updateOne({ _id: "orders" }, { $inc: { n: 1 } });

// Worker B: 100 transactions doing the same, retrying on TransientTransactionError
const s = db.getMongo().startSession();
const c = s.getDatabase("ch22").counters;
let retries = 0;
for (let i = 0; i < 100; i++) {
  while (true) {
    try {
      s.startTransaction();
      c.updateOne({ _id: "orders" }, { $inc: { n: 1 } });
      s.commitTransaction();
      break;
    } catch (e) {
      if (!(e.errorLabels || []).includes("TransientTransactionError")) throw e;
      try { s.abortTransaction(); } catch (_) {}
      retries++;
    }
  }
}
Result
plain worker: 301 ms
plain worker: 314 ms
plain worker: 290 ms
plain worker: 304 ms
plain worker: 289 ms
plain worker: 309 ms
plain worker: 297 ms
plain worker: 310 ms
{ _id: 'orders', n: 800 }
txn worker: 670 ms, 91 retries
txn worker: 730 ms, 108 retries
txn worker: 784 ms, 124 retries
txn worker: 776 ms, 127 retries
txn worker: 803 ms, 133 retries
txn worker: 827 ms, 144 retries
txn worker: 813 ms, 139 retries
txn worker: 831 ms, 150 retries
{ _id: 'orders', n: 1600 }
What’s happening
  1. Eight shells run worker A at the same time. Each $inc is one atomic single-document write; concurrent writers to the same document simply queue for a moment. All eight finish in about 300 ms and the counter is exactly 800.
  2. Then eight shells run worker B at the same time. Each bump opens a transaction, updates the counter and commits.
  3. Whenever two transactions touch the counter at once, the second gets a WriteConflict and is aborted (as in the previous topic), so the worker aborts and tries again. That happened 91–150 times per worker, 1,016 wasted attempts for 800 useful ones.
  4. Each worker took 670–831 ms, about 2.5 times longer, for the same result (n is now 1600: both runs counted correctly).
  5. The transaction bought nothing here, because the change was a single document all along. Hot documents plus transactions turn contention into retries; plain atomic updates turn it into a short queue.
AdvancedWhat a long transaction really blocks

An open transaction blocks writers to the documents it changed, not readers. Here a transaction has updated the charger and not committed yet; a plain write to the same document is allowed to wait 2 seconds:

mongosh
const s = db.getMongo().startSession();
s.startTransaction();
s.getDatabase("ch22").products.updateOne({ _id: 7 }, { $inc: { stock: -1 } })
// Outside the transaction, a plain write to the same document, allowed to wait 2 s:
const t1 = Date.now();
db.runCommand({ update: "products", updates: [{ q: { _id: 7 }, u: { $inc: { stock: 5 } } }], maxTimeMS: 2000 })
print("write waited " + (Date.now() - t1) + " ms")
// A plain read of the same document does not wait:
const t2 = Date.now();
db.products.findOne({ _id: 7 }, { stock: 1 })
print("read took " + (Date.now() - t2) + " ms")
s.commitTransaction()
db.products.findOne({ _id: 7 }, { stock: 1 })
Result
{
  acknowledged: true,
  insertedId: null,
  matchedCount: 1,
  modifiedCount: 1,
  upsertedCount: 0
}
Uncaught MongoServerError[MaxTimeMSExpired]: Plan executor error during update :: caused by :: operation exceeded time limit
write waited 2028 ms
{ _id: 7, stock: 100 }
read took 13 ms
{ _id: 7, stock: 99 }
What’s happening
  1. The transaction decrements the charger's stock (100 → 99 inside the transaction) and holds the document's write lock.
  2. The outside update wants the same document. A non-transactional write waits for the transaction to finish (unlike a second transaction, which fails fast). After its maxTimeMS of 2 seconds it gives up with MaxTimeMSExpired.
  3. The outside findOne returns immediately (13 ms) with the last committed value, 100. MVCC means readers aren't blocked and don't see uncommitted data.
  4. After commit, everyone sees 99. The outside $inc: 5 never happened because it timed out.
  5. Without maxTimeMS, that write would simply hang until the transaction ends, which with a slow app could be up to the 60-second limit. That's the real "locks resources" cost.
AdvancedMeasured: one-statement transactions are not much slower

Without contention, the overhead is smaller than people expect. 500 plain updates against 500 transactions that each do one update, both with the default w: "majority", on the same three-member replica set (two runs):

mongosh
const N = 500;
let t = Date.now();
for (let i = 0; i < N; i++) db.products.updateOne({ _id: 7 }, { $inc: { stock: 1 } });
print(`${N} plain updates:              ${Date.now() - t} ms`);
t = Date.now();
const s = db.getMongo().startSession();
const p = s.getDatabase("ch22").products;
for (let i = 0; i < N; i++) { s.startTransaction(); p.updateOne({ _id: 7 }, { $inc: { stock: 1 } }); s.commitTransaction(); }
print(`${N} one-update transactions:    ${Date.now() - t} ms`);
Result
500 plain updates:              1054 ms
500 one-update transactions:    903 ms
500 plain updates:              971 ms
500 one-update transactions:    1070 ms
What’s happening
  1. Each plain update waits for a majority of the replica set before returning, about 2 ms here on one machine.
  2. Each transaction sends two commands (the update, then commitTransaction), but only the commit waits for the majority, so the totals come out about the same.
  3. On a real network the extra round trip per transaction adds latency, and each transaction costs a little server bookkeeping, but the big costs are the ones shown above: conflicts on hot documents, and long-open transactions.
  4. So "transactions are slow" is the wrong reason to avoid them. "This change already fits in one document" and "these documents are hot" are the right reasons.
AdvancedDesign alternatives to a transaction

Before writing startTransaction, check whether one of these fits:

  • Embed: put data that changes together in one document (order + items + totals), see Embedding vs referencing.
  • Conditional update: put the precondition in the filter (stock: { $gte: 1 }) and check modifiedCount.
  • Atomic operators: $inc, $push, $addToSet, $min/$max change a value relative to its current state without a read.
  • Idempotent steps plus a status field: for work that spans services, record progress (status: "reserved" → "charged" → "shipped") and make each step safe to repeat. That's eventual consistency, the "design around it" part of my notes, and it's how you coordinate things a database transaction can't cover anyway, like a payment provider.

#Replica sets

A replica set is a group of mongod processes that hold the same data: one primary takes all the writes, and secondaries copy every change from it and stand ready to take over. If the primary dies, the remaining members hold an election and one secondary becomes the new primary, usually within about 10–12 seconds, without anyone touching anything. From my notes: replica sets are MongoDB's answer to high availability; sharding (later in this chapter) is the answer to horizontal scaling.

The mental model is a newsroom with one editor and two copy-takers. Only the editor (primary) can publish. Every change goes into a running log (the oplog, next topic), and the copy-takers replay that log onto their own copies. If the editor stops answering for 10 seconds, the copy-takers vote and one of them becomes editor. A member needs votes from a majority of the set to win, which is why replica sets have an odd number of voting members: with three, any two can elect; with two, losing either leaves one, which isn't a majority, so nobody can become primary.

From my notes: "creating a replica set on the console", "adding to the replica set" and "rs.status() to check the status". This is a real three-member set, rs22, in Docker: three mongo:9.0 containers named m1, m2 and m3 on a private network, each an ordinary mongod started with the same set name, mongod --replSet rs22 --bind_ip_all. In mongosh connected to m1, initiate the set with one member and add the other two:

mongosh
rs.initiate({ _id: "rs22", members: [{ _id: 0, host: "m1:27017" }] })
rs.add("m2:27017")
rs.add("m3:27017")
rs.status().members.map(m => ({ name: m.name, state: m.stateStr, health: m.health, syncSource: m.syncSourceHost }))
Result
{ ok: 1, ... }
{ ok: 1, ... }
{ ok: 1, ... }
[
  { name: 'm1:27017', state: 'PRIMARY', health: 1, syncSource: '' },
  {
    name: 'm2:27017',
    state: 'SECONDARY',
    health: 1,
    syncSource: 'm1:27017'
  },
  {
    name: 'm3:27017',
    state: 'SECONDARY',
    health: 1,
    syncSource: 'm1:27017'
  }
]
What’s happening
  1. rs.initiate writes the first replica set configuration (set name rs22, one member m1:27017). The member elects itself, since one vote out of one is a majority, and becomes PRIMARY within a second or two. The { ok: 1, ... } lines are trimmed; the real replies also carry $clusterTime and operationTime.
  2. rs.add("m2:27017") changes the configuration to two members. The primary contacts m2, which starts an initial sync: it copies every database from a sync source, then catches up by replaying the oplog.
  3. rs.add("m3:27017") does the same for the third member. Each rs.add bumps the config version; the full rs.status() showed configVersion: 5 on every member, so they all agree on the config.
  4. About 10 seconds later both new members report SECONDARY with syncSource: 'm1:27017': they pull the primary's oplog and apply it. health: 1 means the heartbeats (every 2 seconds, heartbeatIntervalMillis: 2000) are getting through.
  5. The full rs.status() also showed majorityVoteCount: 2 and writeMajorityCount: 2: with three voting members, 2 votes win an election and 2 copies make a write "majority-committed".
AdvancedWrites only go to the primary

Connect directly to a secondary and try to write:

mongosh
// connected directly to m2, a secondary
db.hello().isWritablePrimary
db.products.insertOne({ _id: 99, name: "Test" })
db.products.findOne({ _id: 1 }, { name: 1 })
Result
false
Uncaught MongoServerError[NotWritablePrimary]: not primary
{ _id: 1, name: 'ThinkBook 14' }
What’s happening
  1. db.hello() is the handshake every driver sends; isWritablePrimary: false says this member can't take writes.
  2. The insert fails with NotWritablePrimary: not primary. A secondary only changes its data by applying the primary's oplog, so there's one ordered history.
  3. The read works: with a direct connection to one member, current drivers and mongosh allow reads on a secondary automatically. In the old mongo shell, the same read failed with "not master and slaveOk=false" until you ran rs.slaveOk() (later rs.secondaryOk()).
  4. In an app you don't connect to one member. You list the members, mongodb://m1:27017,m2:27017,m3:27017/?replicaSet=rs22, and the driver finds the primary itself and follows it after a failover.
AdvancedA planned failover with rs.stepDown()

For maintenance (upgrading the primary's machine) you hand over the primary role on purpose:

mongosh
// on m1, the primary
rs.stepDown(60)
db.hello().isWritablePrimary

// a few seconds later, on m2
rs.status().members.map(m => `${m.name}  ${m.stateStr}`)
rs.status().term
Result
{ ok: 1, ... }
false
[ 'm1:27017  SECONDARY', 'm2:27017  PRIMARY', 'm3:27017  SECONDARY' ]
Long('2')
What’s happening
  1. rs.stepDown(60) asks the primary to step down and not stand for election for 60 seconds. It first waits until a secondary has caught up with its oplog, so no acknowledged write is lost.
  2. m1 is now a secondary (isWritablePrimary: false) and closes its client connections, so drivers rediscover the topology.
  3. The other members hold an election straight away (no 10-second wait, since the old primary handed over deliberately) and m2 wins.
  4. The election term went 1 → 2. Every election increments the term; members reject messages from an older term, which is how an old primary finds out it's been replaced.
AdvancedAn unplanned failover: killing the primary

Now the real thing: the primary's container (m2) is killed with docker compose kill m2, and a shell on m3 asks every second who the primary is:

mongosh
const start = Date.now();
for (let i = 0; i < 20; i++) {
  const h = db.hello();
  print(`${((Date.now() - start) / 1000).toFixed(1)}s  primary: ${h.primary ?? "(none)"}`);
  if (h.primary && h.primary !== "m2:27017") break;
  sleep(1000);
}
Result
0.0s  primary: m2:27017
1.0s  primary: m2:27017
2.0s  primary: (none)
3.0s  primary: (none)
4.0s  primary: (none)
5.0s  primary: (none)
6.0s  primary: (none)
7.0s  primary: (none)
8.0s  primary: (none)
9.1s  primary: (none)
10.1s  primary: (none)
11.1s  primary: m3:27017
AdvancedThe election in the m3 log

The m3 log at the same moment:

Output
{"c":"ELECTION","id":4615652,"msg":"Starting an election, since we've seen no PRIMARY in election timeout period","attr":{"electionTimeoutPeriodMillis":10000}}
{"c":"ELECTION","id":21450,"msg":"Election succeeded, assuming primary role","attr":{"term":3}}
{"c":"REPL","id":21331,"msg":"Transition to primary complete; database writes are now permitted","attr":{"term":3}}
What’s happening
  1. For the first second m3 still believes m2 is primary: nothing has told it otherwise yet.
  2. Heartbeats to m2 start failing and m3 marks it unreachable, so hello() reports no primary. This is the window where writes fail: nobody can accept them.
  3. After electionTimeoutMillis (10 seconds) without a primary, m3 starts an election, gets its own vote plus m1's (2 of 3, a majority) and becomes primary in term 3.
  4. About 11 seconds from crash to a writable primary on this setup, which matches the documented expectation. Drivers with retryable writes (on by default) retry a failed write once after they find the new primary, so many apps ride through it.
  5. When m2 was started again, it rejoined as SECONDARY and caught up from the oplog: [ 'm1:27017 SECONDARY', 'm2:27017 SECONDARY', 'm3:27017 PRIMARY' ].
AdvancedPriority takeover with rs.reconfig

rs.conf() holds each member's settings. Raising a member's priority makes it the preferred primary:

mongosh
const cfg = rs.conf();
cfg.members[0].priority = 2;     // m1 should be primary whenever it is healthy
rs.reconfig(cfg).ok
Result
1
AdvancedOther member options: hidden, delayed, arbiters

About 12 seconds later m1 logged Starting an election for a priority takeover and Election succeeded, assuming primary role (term 4), and rs.status() showed m1 PRIMARY. The other options you'll be asked about:

  • priority 0: never becomes primary (a member in a far-away data centre).
  • hidden: true (needs priority 0): invisible to clients, used for backups or reporting.
  • secondaryDelaySecs (was slaveDelay before 5.0): a hidden member that applies the oplog an hour behind, a safety net against "someone dropped the collection".
  • votes: 0: a non-voting member. A set can have up to 50 members but at most 7 voting ones.
  • Arbiter: votes in elections but holds no data. It lets a two-data-node set elect a primary, but it weakens w: "majority" (see the next topics), so prefer three data-bearing members.

#The oplog

The oplog (operations log) is how replication actually works. It's a special capped collection, local.oplog.rs, on every replica set member. Each write the primary makes is recorded there as one entry; secondaries pull new entries from their sync source and apply them in order. From my notes: "OPLOG, used in a replica set".

The key design choice is that oplog entries are idempotent: applying the same entry twice gives the same result as applying it once. So the primary doesn't log what you asked for ($inc: { stock: -2 }), it logs the result (set stock to 5). That lets a secondary safely re-apply entries after a restart or during initial sync without double-counting. Think of it as a ship's log that records "position now 43°N" rather than "turned 5° left".

mongosh
db.products.insertOne({ _id: 9, name: "Kindle", category: "Books", price: 159, stock: 7 })
db.products.updateOne({ _id: 9 }, { $inc: { stock: -2 } })
db.products.deleteOne({ _id: 9 })
use local
db.oplog.rs.find(
  { ns: "ch22.products", $or: [{ "o._id": 9 }, { "o2._id": 9 }] },
  { _id: 0, op: 1, o: 1, o2: 1, ts: 1 }
).sort({ $natural: -1 }).limit(3)
Result
{ acknowledged: true, insertedId: 9 }
{
  acknowledged: true,
  insertedId: null,
  matchedCount: 1,
  modifiedCount: 1,
  upsertedCount: 0
}
{ acknowledged: true, deletedCount: 1 }
switched to db local
[
  { op: 'd', o: { _id: 9 }, ts: Timestamp({ t: 1792164731, i: 3 }) },
  {
    op: 'u',
    o: { '$v': 2, diff: { u: { stock: 5 } } },
    o2: { _id: 9 },
    ts: Timestamp({ t: 1792164731, i: 2 })
  },
  {
    op: 'i',
    o: { _id: 9, name: 'Kindle', category: 'Books', price: 159, stock: 7 },
    o2: { _id: 9 },
    ts: Timestamp({ t: 1792164731, i: 1 })
  }
]
What’s happening
  1. Three ordinary writes run on the primary: insert the Kindle, decrement its stock by 2, delete it.
  2. local.oplog.rs is queried newest first ($natural: -1 is the order entries were written). Each entry has op (i insert, u update, d delete, c command, n no-op), the namespace ns, the change o, and for updates and inserts the target o2.
  3. The insert entry carries the full document. The delete entry carries only { _id: 9 }, which is all a secondary needs.
  4. The update entry is the interesting one: $inc: { stock: -2 } was recorded as { '$v': 2, diff: { u: { stock: 5 } } }, a version-2 "diff" that says set stock to 5. Applying it twice still leaves 5.
  5. ts is a Timestamp of seconds plus a counter within that second (i: 1, 2, 3). It totally orders the primary's writes, and secondaries record how far they've applied by this value, which is how replication lag is measured.
AdvancedOne statement, many entries; one transaction, one entry

An updateMany is logged as one entry per document, while a transaction is logged as one applyOps command holding all its writes:

mongosh
db.products.updateMany({ category: "Books" }, { $inc: { stock: 10 } })
const s = db.getMongo().startSession();
s.startTransaction();
s.getDatabase("ch22").products.updateOne({ _id: 3 }, { $inc: { stock: -1 } })
s.getDatabase("ch22").orders.insertOne({ _id: 10, customerId: 5, status: "paid", items: [{ productId: 3, qty: 1, price: 799 }] })
s.commitTransaction()
use local
db.oplog.rs.find({ ns: "ch22.products", op: "u" }, { _id: 0, op: 1, o: 1, o2: 1 }).sort({ $natural: -1 }).limit(2)
db.oplog.rs.find({ "o.applyOps": { $exists: true } },
  { _id: 0, op: 1, ns: 1, "o.applyOps.op": 1, "o.applyOps.ns": 1, "o.applyOps.o": 1, lsid: 1, txnNumber: 1 }
).sort({ $natural: -1 }).limit(1)
Result
switched to db local
[
  { op: 'u', o: { '$v': 2, diff: { u: { stock: 45 } } }, o2: { _id: 6 } },
  { op: 'u', o: { '$v': 2, diff: { u: { stock: 60 } } }, o2: { _id: 5 } }
]
[
  {
    lsid: {
      id: UUID('45ecc230-20d5-4529-9956-eb52e8ff353b'),
      uid: Binary.createFromBase64('47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=', 0)
    },
    txnNumber: Long('1'),
    op: 'c',
    ns: 'admin.$cmd',
    o: {
      applyOps: [
        {
          op: 'u',
          ns: 'ch22.products',
          o: { '$v': 2, diff: { u: { stock: 19 } } }
        },
        {
          op: 'i',
          ns: 'ch22.orders',
          o: {
            _id: 10,
            customerId: 5,
            status: 'paid',
            items: [ { productId: 3, qty: 1, price: 799 } ]
          }
        }
      ]
    }
  }
]
What’s happening
  1. The updateMany matched the two books, so the oplog got two separate update entries, each with the resulting stock (35 + 10 = 45, 50 + 10 = 60). A secondary applies them one at a time, which is another way to see that updateMany is not one atomic unit.
  2. The transaction wrote to two collections, yet the oplog holds one entry: op: 'c' (a command) on admin.$cmd, whose applyOps array contains both writes.
  3. Because the whole transaction is one entry, a secondary applies it all-or-nothing, so readers on a secondary never see the order without the stock change.
  4. The entry also records the session id (lsid) and txnNumber. A transaction larger than 16 MB is split over several entries chained together (since 4.2); each entry is still at most 16 MB.
AdvancedOplog size and the oplog window

Two shell helpers summarise the oplog and each secondary's lag:

mongosh
rs.printReplicationInfo()
rs.printSecondaryReplicationInfo()
Result
actual oplog size
'31361.531005859375 MB'
---
configured oplog size
'31361.531005859375 MB'
---
log length start to end
'562 secs (0.16 hrs)'
---
oplog first event time
'Fri Oct 16 2026 15:22:53 GMT+0000 (Coordinated Universal Time)'
---
oplog last event time
'Fri Oct 16 2026 15:32:15 GMT+0000 (Coordinated Universal Time)'
---
now
'Fri Oct 16 2026 15:32:17 GMT+0000 (Coordinated Universal Time)'
source: m2:27017
{
  syncedTo: 'Fri Oct 16 2026 15:32:15 GMT+0000 (Coordinated Universal Time)',
  replLag: '0 secs (0 hrs) behind the primary '
}
---
source: m3:27017
{
  syncedTo: 'Fri Oct 16 2026 15:32:15 GMT+0000 (Coordinated Universal Time)',
  replLag: '0 secs (0 hrs) behind the primary '
}
What’s happening
  1. The configured size is about 31 GB. With WiredTiger the default is 5% of free disk space, at least 990 MB and at most 50 GB; the Docker VM here has about 630 GB free.
  2. "log length start to end" is the oplog window: the time between the oldest and newest entry, only 562 seconds because this set is ten minutes old. On a busy production set, the window is what matters: a secondary that's offline longer than the window can't catch up from the oplog and needs a full initial sync.
  3. printSecondaryReplicationInfo shows each secondary's last applied operation and its replication lag, 0 secs here. Lag is what you alert on.
  4. The oplog is allowed to grow past its configured size when needed to avoid deleting the majority commit point, which no other capped collection does.
AdvancedResizing the oplog on a running member

You can resize it while running, and set a minimum retention time (since 4.4) so entries are kept for at least that long:

mongosh
db.adminCommand({ replSetResizeOplog: 1, size: 2048, minRetentionHours: 24 }).ok
db.getSiblingDB("local").oplog.rs.stats().maxSize
db.adminCommand({ replSetResizeOplog: 1, size: 500 })
Result
1
2147483648
Uncaught MongoServerError[BadValue]: BSON field 'size' value must be >= 990, actual value '500'
AdvancedResize rules and other oplog uses

The size is in megabytes (2048 MB = 2,147,483,648 bytes), the minimum is the same 990 MB as the default floor, and the change applies to the member you run it on, so you run it on each member and also put it in the config file (replication.oplogSizeMB, storage.oplogMinRetentionHours) so it survives restarts.

  • Change streams (db.collection.watch()) are built on the oplog: they let an app react to inserts, updates and deletes, with a resume token so it can pick up where it left off, as long as that point is still in the oplog.
  • Point-in-time backups: mongodump --oplog captures the oplog entries written during the dump, and mongorestore --oplogReplay replays them (see Backup and restore).
  • Initial sync copies the data, then replays the oplog entries written during the copy; idempotency is what makes replaying "maybe already applied" entries safe.
  • Never write to local.oplog.rs yourself. Reading it is fine for learning; apps should use change streams instead.

#Write concern

Write concern is how many members must confirm a write before the server tells your app "done". It's a dial between speed and safety, set per operation, per connection or as a cluster default. From my notes: w: 1, w: "majority", w: 0 and j: true (journaling).

  • w: 0: don't wait for anything. The driver sends the write and moves on; you don't even learn about errors.
  • w: 1: the primary has applied it (in memory).
  • w: "majority": a majority of data-bearing voting members have it. The default since MongoDB 5.0.
  • w: <n>: that many members have it.
  • j: true: the acknowledging members have also written it to their on-disk journal, so it survives a crash of that machine.
  • wtimeout: how long to wait for the w part before reporting an error.

The mental model is a courier. w: 0 is dropping the letter in a postbox. w: 1 is a signature from the front desk. w: "majority" is signatures from two of the three offices, so the letter survives even if the front desk burns down.

mongosh
db.products.insertOne({ _id: 20, name: "Mouse" },    { writeConcern: { w: 0 } })
db.products.insertOne({ _id: 21, name: "Keyboard" }, { writeConcern: { w: 1 } })
db.products.insertOne({ _id: 22, name: "Monitor" },  { writeConcern: { w: "majority" } })
db.products.insertOne({ _id: 23, name: "Webcam" },   { writeConcern: { w: "majority", j: true, wtimeout: 5000 } })
db.products.insertOne({ _id: 24, name: "Dock" },     { writeConcern: { w: 3 } })
db.products.insertOne({ _id: 25, name: "Hub" },      { writeConcern: { w: 4, wtimeout: 2000 } })
db.products.countDocuments({ _id: { $gte: 20 } })
Result
{ acknowledged: false, insertedId: 20 }
{ acknowledged: true, insertedId: 21 }
{ acknowledged: true, insertedId: 22 }
{ acknowledged: true, insertedId: 23 }
{ acknowledged: true, insertedId: 24 }
Uncaught:
MongoWriteConcernError[UnsatisfiableWriteConcern]: Not enough data-bearing nodes
Additional information: {
  writeConcern: { w: 4, wtimeout: 2000, provenance: 'clientSupplied' }
}
Result: {
  n: 1,
  electionId: ObjectId('7fffffff0000000000000004'),
  opTime: { ts: Timestamp({ t: 1792164757, i: 6 }), t: 4 },
  writeConcernError: {
    code: 100,
    codeName: 'UnsatisfiableWriteConcern',
    errmsg: 'Not enough data-bearing nodes',
    errInfo: {
      writeConcern: { w: 4, wtimeout: 2000, provenance: 'clientSupplied' }
    }
  },
  ok: 1,
  ...
}
6
What’s happening
  1. w: 0 returns acknowledged: false: the shell didn't wait for any reply, so it can't tell you whether the insert worked. (insertedId is known only because the client generated it.)
  2. w: 1, w: "majority" and w: 3 all return acknowledged: true, but they waited for different things: the primary only, two of three members, all three members. On this healthy set the difference is a millisecond or two.
  3. j: true with w: "majority" makes the journal explicit. With "majority" it's already the default behaviour, because writeConcernMajorityJournalDefault is true; with w: 1 and no j, the primary acknowledges once the write is in memory.
  4. w: 4 can never be satisfied with three data-bearing members, so the server returns a write concern error, UnsatisfiableWriteConcern. Note n: 1 and ok: 1: the document was written; only the confirmation you asked for failed.
  5. countDocuments returns 6, all six inserts including the "failed" one. A write concern error is never a rollback; it means "not confirmed as you asked", and your app has to decide what to do about it.
AdvancedProved: w:1 writes can be rolled back

This is why the default changed to "majority". The primary m1 is cut off from the other two with docker network disconnect, and before it notices, an app writes two orders to it:

mongosh
// on m1, which has just lost contact with m2 and m3 but doesn't know it yet
db.hello().isWritablePrimary
db.orders.insertOne({ _id: 40, customerId: 6, status: "paid", total: 249 }, { writeConcern: { w: 1 } })
db.orders.insertOne({ _id: 41, customerId: 6, status: "paid", total: 29 }, { writeConcern: { w: "majority", wtimeout: 3000 } })
Result
true
{ acknowledged: true, insertedId: 40 }
Uncaught:
MongoWriteConcernError[WriteConcernTimeout]: waiting for replication timed out
Additional information: {
  wtimeout: true,
  writeConcern: { w: 'majority', wtimeout: 3000, provenance: 'clientSupplied' }
}
AdvancedThe isolated primary rolls back

Meanwhile m2 and m3 elected m2 as primary (term 5) and took a new order, _id: 42. Then the network was reconnected. m1 rejoined, found that its history had diverged, and rolled back:

Output
{"c":"ROLLBACK","id":21612,"msg":"Rollback summary","attr":{"syncSource":"m2:27017","rollbackDataFileDirectory":"/data/db/rollback/3f2b9255-7f19-42cb-ac13-ae4caf3397b6", ... "rollbackCommandCounts":{"insert":2} ...}}
AdvancedWhich orders survived the rollback

Then, which of the three orders still exist?

mongosh
db.orders.find({ _id: { $in: [40, 41, 42] } }, { total: 1 }).toArray()
Result
[ { _id: 42, total: 39 } ]
AdvancedReading the rollback file

The rolled-back documents were saved to a BSON file, readable with bsondump:

Output
{"_id":{"$numberInt":"41"},"customerId":{"$numberInt":"6"},"status":"paid","total":{"$numberInt":"29"}}
{"_id":{"$numberInt":"40"},"customerId":{"$numberInt":"6"},"status":"paid","total":{"$numberInt":"249"}}
What’s happening
  1. Cut off from the others, m1 still believes it's primary for a few seconds (isWritablePrimary: true). It takes both inserts.
  2. Order 40 used w: 1, so the app got acknowledged: true and would have shown the customer "Order placed".
  3. Order 41 used w: "majority". No other member could confirm it, so after 3 seconds the app got WriteConcernTimeout. The app knows this order isn't safe.
  4. m2 and m3 still form a majority, so they elect m2 and carry on; order 42 is written there.
  5. When m1 reconnects it sees a higher term, becomes a secondary, and finds oplog entries (orders 40 and 41) that the new primary never had. It rolls them back to the common point and writes them to /data/db/rollback/... for a human to deal with.
  6. Order 40 is gone from the database even though the app was told it succeeded. With w: "majority", an acknowledged write can't be lost this way, because any member that can win an election must have it.
AdvancedA majority that can't be reached

To see "majority" wait for real, both secondaries are frozen with db.fsyncLock(): they stay up and answer heartbeats (so m1 stays primary) but apply nothing:

mongosh
// on m2 and on m3
db.fsyncLock().info

// on m1, the primary
db.products.insertOne({ _id: 30, name: "Tablet" }, { writeConcern: { w: 1 } })
db.products.insertOne({ _id: 31, name: "Stylus" }, { writeConcern: { w: "majority", wtimeout: 3000 } })
db.products.find({ _id: { $in: [30, 31] } }, { name: 1 })
Result
now locked against writes, use db.fsyncUnlock() to unlock
now locked against writes, use db.fsyncUnlock() to unlock
{ acknowledged: true, insertedId: 30 }
Uncaught:
MongoWriteConcernError[WriteConcernTimeout]: waiting for replication timed out
Additional information: {
  wtimeout: true,
  writeConcern: { w: 'majority', wtimeout: 3000, provenance: 'clientSupplied' }
}
Result: {
  n: 1,
  ...
  writeConcernError: {
    code: 64,
    codeName: 'WriteConcernTimeout',
    errmsg: 'waiting for replication timed out',
    ...
  },
  ok: 1,
  ...
}
[ { _id: 30, name: 'Tablet' }, { _id: 31, name: 'Stylus' } ]
What’s happening
  1. w: 1 returns immediately: the primary alone is enough.
  2. w: "majority" needs one more member to apply the write; both are frozen, so after wtimeout: 3000 the server returns WriteConcernTimeout (code 64).
  3. Again n: 1: the Stylus is on the primary, and a plain read shows both documents.
  4. Without wtimeout, the insert would have waited forever. Set wtimeout (or a driver timeout such as timeoutMS) on majority writes so a sick replica set produces errors instead of hung requests.
  5. The next topic uses this same frozen state to show what different read concerns return.
AdvancedEdge case: w:0 also hides errors

A write the server rejects looks like a success with w: 0:

mongosh
db.products.insertOne({ _id: 1, name: "Duplicate!" }, { writeConcern: { w: 0 } })
db.products.insertOne({ _id: 1, name: "Duplicate!" }, { writeConcern: { w: 1 } })
db.products.findOne({ _id: 1 }, { name: 1 })
Result
{ acknowledged: false, insertedId: 1 }
Uncaught MongoServerError: E11000 duplicate key error collection: ch22.products index: _id_ dup key: { _id: 1 }
{ _id: 1, name: 'ThinkBook 14' }
Advancedw:0 in practice and the cluster default

Both inserts fail on the server (product 1 already exists), but with w: 0 the app never hears about it. It's only defensible for data you can afford to lose, such as high-volume metrics, and even there w: 1 usually costs little.

The cluster-wide default comes from getDefaultRWConcern:

mongosh
db.adminCommand({ getDefaultRWConcern: 1 })
Result
{
  defaultReadConcern: { level: 'local' },
  defaultWriteConcern: { w: 'majority', wtimeout: 0 },
  defaultWriteConcernSource: 'implicit',
  defaultReadConcernSource: 'implicit',
  ...
}
AdvancedDefaults, arbiters and older versions

implicit means nobody set it; you can set one with setDefaultRWConcern. Things to know around it:

  • Before 5.0 the implicit default was w: 1. Apps upgraded from 4.x got majority writes automatically, slightly slower but safer.
  • Arbiters change the default. If a set has arbiters and the data-bearing members aren't more than a majority of voters (the classic primary + secondary + arbiter), the implicit default is w: 1, because with one data node down "majority" could never be satisfied.
  • Very old drivers (before 2012) defaulted to fire-and-forget (w: 0), the source of many "MongoDB loses data" stories. Modern drivers acknowledge by default.
  • getLastError, the old way to ask "did my last write work?", is gone; the write concern goes on the write itself.
  • In a transaction you set the write concern on the transaction (startTransaction({ writeConcern })), not on its individual operations.

#Read concern and read preference

Added

Write concern controls how safe a write is before you're told it worked. The read side has two separate dials. Read concern decides which version of the data a read may return: anything the node has ("local"), or only data a majority has confirmed ("majority"), which can't be rolled back. Read preference decides which member serves the read: the primary, or a secondary that may be behind.

The mental model: read concern is "how confirmed must the news be before you tell me?"; read preference is "which newsroom should I ask?". You can combine them: majority-committed data from the nearest secondary is a valid request.

The main example reuses the frozen secondaries from the last topic: products 30 and 31 exist on the primary only.

mongosh
// connected with mongodb://m1:27017,m2:27017,m3:27017/ch22?replicaSet=rs22
db.products.find({ _id: { $in: [30, 31] } }, { name: 1 }).readConcern("local").toArray()
db.products.find({ _id: { $in: [30, 31] } }, { name: 1 }).readConcern("majority").toArray()
db.products.find({ _id: { $in: [30, 31] } }, { name: 1 }).readPref("secondary").toArray()
Result
[ { _id: 30, name: 'Tablet' }, { _id: 31, name: 'Stylus' } ]
[]
[]
What’s happening
  1. readConcern("local") (the default) on the primary returns whatever the primary has, including the two documents no other member has confirmed. These are exactly the kind of writes that were rolled back in the previous topic.
  2. readConcern("majority") on the same primary returns nothing: it reads from the majority-committed snapshot, and those documents haven't reached a majority. Whatever it does return can never be rolled back.
  3. readPref("secondary") sends the read to m2 or m3. They're frozen, so they don't have the documents yet: a stale read, the normal price of reading from secondaries.
  4. After db.fsyncUnlock() on both secondaries, the same "majority" read and the same secondary read both returned [ { _id: 30, name: 'Tablet' }, { _id: 31, name: 'Stylus' } ].
  5. So "majority" is about durability of what you read; read preference is about freshness and load. Neither guarantees the other.
AdvancedLinearizable reads cost a round of confirmation

"linearizable" (primary only, single-document reads) promises the read reflects every write that was majority-acknowledged before it started, even across a failover. To keep that promise, the primary confirms it's still primary with a majority before answering. With the secondaries frozen:

mongosh
db.products.find({ _id: 30 }, { name: 1 }).readConcern("linearizable").maxTimeMS(3000).toArray()
Result
Uncaught MongoServerError[MaxTimeMSExpired]: operation exceeded time limit
What’s happening
  1. A linearizable read on the primary writes a no-op to the oplog and waits until a majority has it, which proves no other member has been elected in the meantime.
  2. With both secondaries frozen, that confirmation can't happen, so the read waits until maxTimeMS (3 seconds) and fails.
  3. After db.fsyncUnlock() on both secondaries, the same read returned [ { _id: 30, name: 'Tablet' } ] at once.
  4. With a healthy set it costs one replication round trip per read. Use it for the rare reads that must never be stale, such as a lock or a balance check, always with maxTimeMS.
AdvancedThe read preference modes
ModeReads fromTypical use
primary (default)the primary only; fails if there is nonenormal app reads, read-your-own-writes
primaryPreferredthe primary, or a secondary during a failoverstay readable during elections, accepting stale reads
secondarysecondaries onlyreporting and analytics, keeping load off the primary
secondaryPreferredsecondaries, or the primary if none are availablethe same, but still works on a single-member set
nearestthe member with the lowest network latency, primary or secondarymulti-region apps reading from the local data centre

Two refinements: tag sets ({ region: "eu" }) pick members by tags in their config, and maxStalenessSeconds (at least 90) excludes secondaries that are too far behind. Secondary reads don't add write capacity, though: every secondary applies every write too, so they help read-heavy loads, not write-heavy ones.

AdvancedCausal consistency: read your own writes

A user updates their profile, then the next page reads it from a secondary that hasn't caught up, and the old value appears. A causally consistent session fixes that: the driver remembers the cluster time of your last write and sends it with the next read (afterClusterTime), and the secondary waits until it has applied up to that point before answering. In drivers you get it with client.startSession({ causalConsistency: true }) (the default for explicit sessions) combined with "majority" read and write concern. It only applies to operations that use that session.

#Sharding

A replica set copies all the data to every member, so it can't hold more data, or take more writes, than one machine can. Sharding splits a collection's documents across several replica sets, called shards, by the value of a shard key. From my notes: "Horizontal scaling: Sharding". Each shard is a full replica set (so you keep high availability), and together they look like one database to the app.

A sharded cluster has three parts:

  • Shards: replica sets that each hold a subset of the data.
  • Config servers: a replica set that stores the metadata, which range of shard key values (a chunk) lives on which shard.
  • mongos: the query router. Apps connect to mongos, never to shards directly. It reads the metadata and sends each operation to the right shard or shards.

The mental model is a library split across several buildings by call number. The catalogue (config servers) says which building holds which range; the front desk (mongos) takes your request, looks at the call number (the shard key) and walks to one building, or, if you ask for "every book with a red cover", has to search all of them.

This one ran for real: a one-node config server replica set, two one-node shards (sh1, sh2) and a mongos, all mongo:9.0 containers. In production each would be a three-member replica set on its own machines. The orders collection is sharded on a hashed customerId, then loaded with 20,000 orders for 1,000 customers:

mongosh
// each shard was started with: mongod --shardsvr --replSet sh1 (and sh2), then rs.initiate()
// the config server with: mongod --configsvr --replSet cfgrs, the router with: mongos --configdb cfgrs/cfg:27017
// connected to mongos:
sh.addShard("sh1/shard1:27017")
sh.addShard("sh2/shard2:27017")
sh.shardCollection("ch22.orders", { customerId: "hashed" })

const statuses = ["pending", "paid", "shipped", "cancelled"];
const docs = [];
for (let i = 1; i <= 20000; i++)
  docs.push({ _id: i, customerId: (i % 1000) + 1, status: statuses[i % 4], total: (i % 50) * 10 + 9 });
db.orders.insertMany(docs).insertedIds.length
db.orders.getShardDistribution()
Result
{ shardAdded: 'sh1', ok: 1, ... }
{ shardAdded: 'sh2', ok: 1, ... }
{ collectionsharded: 'ch22.orders', ok: 1, ... }
20000
Shard sh1 at sh1/shard1:27017
{
  data: '617KiB',
  docs: 10400,
  chunks: 1,
  'estimated data per chunk': '617KiB',
  'estimated docs per chunk': 10400
}
---
Shard sh2 at sh2/shard2:27017
{
  data: '568KiB',
  docs: 9600,
  chunks: 1,
  'estimated data per chunk': '568KiB',
  'estimated docs per chunk': 9600
}
---
Totals
{
  data: '1.15MiB',
  docs: 20000,
  chunks: 2,
  'Shard sh1': [
    '52.05 % data',
    '52 % docs in cluster',
    '60B avg obj size on shard'
  ],
  'Shard sh2': [
    '47.94 % data',
    '48 % docs in cluster',
    '60B avg obj size on shard'
  ]
}
What’s happening
  1. sh.addShard registers each shard replica set with the config servers, using the setName/host form so mongos can find the shard's primary.
  2. sh.shardCollection("ch22.orders", { customerId: "hashed" }) creates the index on customerId (hashed), records the shard key, and, because the collection is empty and the key is hashed, pre-creates chunks covering the whole hash range and spreads them over both shards: one chunk each here.
  3. insertMany goes through mongos. For each document it hashes customerId, finds which chunk that hash falls into, and sends the insert to that chunk's shard. The app does nothing special.
  4. getShardDistribution shows the result: 10,400 documents on sh1 and 9,600 on sh2, about 52/48. Hashing turned 1,000 customer ids into well-spread values, so each shard gets roughly half the data and half the writes.
  5. Each shard holds only its part; add a third shard and the balancer moves chunks so it takes a third. That's horizontal scaling: capacity grows by adding machines, not by buying a bigger one.
AdvancedTargeted queries vs scatter-gather

mongos can only send a query to one shard if the query includes the shard key. Otherwise it has to ask every shard and merge the answers:

mongosh
const a = db.orders.find({ customerId: 42 }).explain("executionStats");
({ stage: a.queryPlanner.winningPlan.stage, shards: a.queryPlanner.winningPlan.shards.map(s => s.shardName),
   nReturned: a.executionStats.nReturned })
const b = db.orders.find({ status: "cancelled" }).explain("executionStats");
({ stage: b.queryPlanner.winningPlan.stage, shards: b.queryPlanner.winningPlan.shards.map(s => s.shardName),
   nReturned: b.executionStats.nReturned })
Result
{ stage: 'SINGLE_SHARD', shards: [ 'sh2' ], nReturned: 20 }
{ stage: 'SHARD_MERGE', shards: [ 'sh1', 'sh2' ], nReturned: 5000 }
What’s happening
  1. customerId: 42 contains the shard key, so mongos hashes 42, looks up the one chunk that covers it, and sends the query only to sh2: SINGLE_SHARD, 20 orders.
  2. status: "cancelled" says nothing about customerId, so mongos can't know where the documents are. It sends the query to both shards and merges the results: SHARD_MERGE, 5,000 orders.
  3. With two shards that's harmless. With 50 shards, every such query costs 50 shard queries, and adding shards stops helping. Your most frequent queries should include the shard key.
  4. For big analytical aggregations, scatter-gather is fine, even useful, because every shard works on its part in parallel.
AdvancedChunks, the balancer and recent changes
  • Chunks are contiguous ranges of shard key values. The default range size is 128 MB (it was 64 MB before 6.0).
  • The balancer runs on the config server primary and moves ranges between shards when the data size difference between shards passes a threshold (since 6.0.3 it balances by data size, not chunk count). A shard takes part in one migration at a time, and migrations cost I/O, so they're scheduled into a balancing window on busy clusters.
  • A chunk that can't be split because it holds a single shard key value is a jumbo chunk; that's a shard-key problem (next topic).
  • 8.0 added config shards (the config server can also hold application data, handy for small clusters), moveCollection (move an unsharded collection to another shard), unshardCollection, and sh.shardAndDistributeCollection() to shard and spread existing data quickly by resharding to the same key.
  • Unsharded collections live on their database's primary shard. Sharding is per collection, so small collections usually stay unsharded.
  • Not run here: a production-shaped cluster (three-member shards and config servers on separate machines, several mongos), and balancing under real load. The cluster above is the minimum that shows routing and distribution.

#Choosing a shard key

Added

The shard key is the most important decision in a sharded cluster, because it decides where every document lives, which queries can be targeted, and whether writes spread out or pile onto one shard. Since 5.0 you can change it with reshardCollection, but that rewrites the whole collection, so you want to get it roughly right first.

A good shard key has three properties, and the official guidance names them exactly:

  • High cardinality: many distinct values. Each value lives in exactly one chunk, so a key with 7 distinct values can never use more than 7 chunks.
  • Low frequency: no single value is very common. If 40% of documents share one value, that value's chunk becomes huge and unsplittable, a jumbo chunk.
  • Not monotonically increasing: if each new value is bigger than the last (a timestamp, an auto-incremented id, an ObjectId), every insert lands in the top chunk, on one shard.

And one property about your app: your common queries include it, so they can be targeted. The mental model: you're choosing how to file mail into post-office boxes. File by first letter of surname and the boxes fill evenly. File by year of birth and all the new babies go into the last box.

The main example shows the monotonic problem. events is sharded by range on createdAt, split at 1 October with the older range moved to sh2, so it looks like a cluster that has been running for a while. Then a day of new events arrives:

mongosh
sh.shardCollection("ch22.events", { createdAt: 1 }).ok
sh.splitAt("ch22.events", { createdAt: ISODate("2026-10-01") }).ok
db.adminCommand({ moveChunk: "ch22.events", find: { createdAt: ISODate("2026-09-01") }, to: "sh2" }).ok
// a day of new events, one every 10 seconds
const start = ISODate("2026-10-16T00:00:00Z").getTime();
const ev = [];
for (let i = 0; i < 8640; i++) ev.push({ createdAt: new Date(start + i * 10000), type: "page_view" });
db.events.insertMany(ev).insertedIds.length
db.events.getShardDistribution()
Result
1
1
1
8640
Shard sh2 at sh2/shard2:27017
{
  data: '0B',
  docs: 0,
  chunks: 1,
  'estimated data per chunk': '0B',
  'estimated docs per chunk': 0
}
---
Shard sh1 at sh1/shard1:27017
{
  data: '514KiB',
  docs: 8640,
  chunks: 1,
  'estimated data per chunk': '514KiB',
  'estimated docs per chunk': 8640
}
---
Totals
{
  data: '514KiB',
  docs: 8640,
  chunks: 2,
  'Shard sh2': [ '0 % data', '0 % docs in cluster', '0B avg obj size on shard' ],
  'Shard sh1': [
    '100 % data',
    '100 % docs in cluster',
    '61B avg obj size on shard'
  ]
}
What’s happening
  1. A ranged shard key on createdAt starts as one chunk from MinKey to MaxKey. splitAt cuts it at 1 October, and moveChunk sends the lower chunk (MinKey to 1 Oct) to sh2. Now sh1 owns 1 Oct to MaxKey.
  2. Every new event has a timestamp of "now", 16 October, which is always greater than any boundary. All 8,640 inserts land in the chunk ending at MaxKey, on sh1.
  3. sh2 gets 0% of the writes. Add ten more shards and they'd get 0% too: the write load can never be spread with this key, because new data is always at the top end.
  4. The balancer would eventually split and move the top chunk, but the newest range is always on one shard, so that shard stays the hot spot. This is the classic monotonic-key mistake, and _id: ObjectId has the same problem because ObjectIds start with a timestamp.
  5. Fixes: hash the key ({ createdAt: "hashed" }, below), or lead with a well-spread field, e.g. { deviceId: 1, createdAt: 1 }, which also keeps one device's events together for range queries.
AdvancedMeasuring candidates with analyzeShardKey

analyzeShardKey (7.0+) reports a candidate key's cardinality, most common values and monotonicity from the actual data. Each candidate needs an index that starts with it:

mongosh
db.orders.createIndex({ status: 1 })
db.orders.createIndex({ customerId: 1 })
const k1 = db.orders.analyzeShardKey({ status: 1 }, { keyCharacteristics: true, readWriteDistribution: false }).keyCharacteristics;
({ numDistinctValues: k1.numDistinctValues, mostCommonValues: k1.mostCommonValues.slice(0, 2), monotonicity: k1.monotonicity.type })
const k2 = db.orders.analyzeShardKey({ customerId: 1 }, { keyCharacteristics: true, readWriteDistribution: false }).keyCharacteristics;
({ numDistinctValues: k2.numDistinctValues, mostCommonValues: k2.mostCommonValues.slice(0, 2), monotonicity: k2.monotonicity.type })
const k3 = db.events.analyzeShardKey({ createdAt: 1 }, { keyCharacteristics: true, readWriteDistribution: false }).keyCharacteristics;
({ numDistinctValues: k3.numDistinctValues, monotonicity: k3.monotonicity })
Result
status_1
customerId_1
{
  numDistinctValues: Long('4'),
  mostCommonValues: [
    { value: { status: 'shipped' }, frequency: Long('5000') },
    { value: { status: 'cancelled' }, frequency: Long('5000') }
  ],
  monotonicity: 'not monotonic'
}
{
  numDistinctValues: Long('1000'),
  mostCommonValues: [
    { value: { customerId: 689 }, frequency: Long('20') },
    { value: { customerId: 49 }, frequency: Long('20') }
  ],
  monotonicity: 'not monotonic'
}
{
  numDistinctValues: Long('8640'),
  monotonicity: { recordIdCorrelationCoefficient: 1, type: 'monotonic' }
}
What’s happening
  1. status has only 4 distinct values, each on 5,000 orders. As a shard key it could never use more than 4 chunks, so never more than 4 shards, and each chunk would grow without limit: low cardinality, high frequency, a bad key.
  2. customerId has 1,000 distinct values, the most common appearing only 20 times: high cardinality, low frequency, not monotonic. A good key, and the app's main query ("this customer's orders") includes it.
  3. createdAt on events is reported monotonic with a correlation coefficient of 1: values rise exactly with insertion order, which confirms the hot spot shown above.
  4. With readWriteDistribution: true (after enabling query sampling with configureQueryAnalyzer), it also reports how many of your real queries each candidate would target to one shard.
AdvancedFixing a bad key with reshardCollection

Since 5.0 you can change the key on a live collection. events is moved onto a hashed createdAt:

mongosh
const t = Date.now();
db.adminCommand({ reshardCollection: "ch22.events", key: { createdAt: "hashed" } }).ok
print("resharding took " + Math.round((Date.now() - t) / 1000) + " s")
db.events.getShardDistribution()
Result
1
resharding took 331 s
Shard sh2 at sh2/shard2:27017
{
  data: '260KiB',
  docs: 4369,
  ...
}
---
Shard sh1 at sh1/shard1:27017
{
  data: '254KiB',
  docs: 4271,
  ...
}
...
What’s happening
  1. reshardCollection builds a new copy of the collection, distributed by the new key, on the shards, while the old one keeps serving reads and writes. It then applies the changes made during the copy, blocks writes briefly and switches over.
  2. After the switch the 8,640 events are split 4,369 / 4,271: hashing spread the timestamps evenly.
  3. It took 331 seconds for half a megabyte, because resharding runs for at least reshardingMinimumOperationDurationMillis, 300,000 ms (5 minutes) by default, on purpose, to pace the copy. On a real collection it needs free disk for a full second copy and takes hours.
  4. The catch with the hashed key: a range query like "events from 9:00 to 10:00" now has to ask every shard, because neighbouring timestamps hash to different chunks. Hashing trades range locality for even writes.
  5. Smaller fixes: refineCollectionShardKey (4.4+) adds suffix fields to an existing key, e.g. { customerId: 1 } → { customerId: 1, orderId: 1 }, to split jumbo chunks without a full reshard.
AdvancedShard key rules worth knowing
  • Shard key values can be changed since 4.2 (unless the key is _id): the update below on mongos moved an order from customer 42 to 43, possibly to another shard. Frequent changes are expensive, because each one moves the document between shards.
  • Documents can be missing shard key fields since 4.4; they're treated as null for routing.
  • Unique indexes on a sharded collection must include the shard key as a prefix (_id is unique per shard only, unless it's the shard key), because no single shard can check uniqueness across all of them.
  • Since 7.1, updateOne, deleteOne and findOneAndUpdate can run on a sharded collection without the full shard key in the filter (earlier versions refused); they still have to search more shards.
mongosh
db.orders.updateOne({ customerId: 42 }, { $set: { customerId: 43 } })
Result
{
  acknowledged: true,
  insertedId: null,
  matchedCount: 1,
  modifiedCount: 1,
  upsertedCount: 0
}
Pick the shard key

A ride-hailing app stores trips: { _id: ObjectId, riderId, driverId, city, startedAt, ... }. 90% of queries are "trips for this rider, newest first". There are 30 cities, 5 million riders, and writes are 20,000 trips a minute. Which shard key: { city: 1 }, { _id: 1 }, { startedAt: "hashed" } or { riderId: 1, startedAt: 1 }?

Show answer

{ riderId: 1, startedAt: 1 }.

What’s happening
  1. { city: 1 } has cardinality 30 and very uneven frequency (one big city could hold 20% of trips). It can't use more than 30 chunks and the big cities become jumbo chunks.
  2. { _id: 1 } with ObjectIds is monotonic: every new trip goes to the top chunk on one shard, so 20,000 writes a minute hit one machine.
  3. { startedAt: "hashed" } spreads writes evenly, but "trips for this rider" doesn't contain the key, so every one of the main queries is scatter-gather.
  4. { riderId: 1, startedAt: 1 } has 5 million leading values (high cardinality, low frequency), isn't monotonic because riders are random, and the main query (riderId plus a sort on startedAt) targets one shard and reads one contiguous range. It's also supported by the key's own index.

SQL and MySQL written fresh; MongoDB built from Rohit’s “mongoDB interview” doc. Every query was run on MySQL 9.7, MongoDB 9.0, Redis 8.10 or PostgreSQL 18.

RohitDownloads · All chapters

Sync progress across devices

Type the same private phrase on your Mac and your phone, and your Learned ticks follow you between them — on every study site.

The phrase never leaves this device: only a fingerprint of it is sent, and the server stores a fingerprint of that. Anyone who knows the phrase could see or change your ticks, so pick something you don’t use elsewhere.

Sync is on in this browser.

Sync ID

This ID must be the same on every device. If another device shows a different one, its phrase is different (capital letters count): tap “Turn off here” on it and type the phrase again exactly.