The moment your design has two services with two separate databases, or even just one database split into shards, the simple "begin, then commit" transaction you've relied on your whole career stops working. A flow like "reserve the inventory, charge the card, create the shipment" now stretches across three separate systems, and any one of them can fail after the others have already succeeded. This lesson covers the four honest answers to that problem, and how to pick between them.

Why you cannot just use a transaction

A transaction on a single database gives you atomicity because that one database owns its own log, its own locks, and the single decision of whether to commit or not. Once you spread that same work across separate services, none of that exists anymore.

• Service A finishes and confirms its work, then service B fails. Now the system is left in a half-finished state a user could actually see.
• Service B is slow, and service A is still holding locks while it waits on a network call to B. That creates contention, and a chain of timeouts spreading outward.
• Either service can crash in the gap between actually doing the work and recording that it did the work.

Say these three principles out loud before proposing any specific mechanism.

The best distributed transaction is the one you never needed in the first place. If two pieces of data absolutely must change together, that's a strong hint they actually belong in the same service and the same database. Proposing to redraw the service boundary is a legitimate, senior-level answer, not a dodge.

Real business processes are already eventually consistent, and that's fine. A hotel takes your booking now, charges your card a little later, and cancels the booking if that payment fails. Real workflows already work by compensating for failure afterward, not by being perfectly atomic. Designing it that way is simply matching reality, not settling for less.

Choose your mechanism based on how long the work takes and how long you can afford to hold a lock. A process that finishes in milliseconds, inside one data center, where atomicity really is non-negotiable, points toward one kind of solution. A workflow spanning several seconds and several services points toward a very different one.

Remember: the best distributed transaction is the one you avoid needing. If two pieces of data must always change together, that's a sign they belong in the same database.

Two-phase commit (2PC)

Picture a wedding officiant asking three witnesses, one after another, "do you agree this marriage should happen?" Each witness who says yes has made a binding promise; once everyone has agreed, none of them can take it back. Only after all three say yes does the officiant announce "married." If the officiant collapses right after hearing the last yes but before making the announcement, all three witnesses are stuck: they've each promised, but nobody has told them whether it's actually final. Two-phase commit works exactly like that officiant, steering every participant toward one single shared decision.

This genuinely gives you real atomicity. Here's what it costs you in return.

The coordinator becomes a single point of failure at the exact worst possible moment. If it crashes after everyone voted yes, but before telling anyone the final decision, every single participant is now stuck, still holding its locks, unable to decide on its own what to do. This is exactly why two-phase commit is often called a blocking protocol.

Locks are held across network round trips, so under real contention, throughput collapses, and one slow participant can stall every other one.

Availability multiplies downward. The whole transaction needs every single participant to be up and reachable at once.

Support for it is also uneven; most modern databases and message brokers simply don't implement the standard needed for it at all.

Use two-phase commit for a small number of participants, inside a single data center, where atomicity truly is non-negotiable and the volume is modest, such as some financial systems or inside certain distributed databases themselves. Do not use it for anything long-running, anything a user is actively waiting on, or anything crossing organizational boundaries.

Three-phase commit adds an extra step before committing, to make the protocol non-blocking in some failure cases. It's rarely used in practice, because it adds an extra network round trip and can still get stuck during a genuine network partition. Modern systems that need this kind of guarantee instead reach for a consensus protocol, like Raft or Paxos, covered in the next lesson, which is built from the ground up to survive failures. Google's Spanner, for instance, actually runs two-phase commit on top of groups running Paxos, so no single coordinator's failure can ever block anything.

Remember: in 2PC, a YES vote is a promise, not a suggestion. If the coordinator dies right after collecting the votes, every participant is stuck holding its lock until the coordinator comes back.

Sagas: compensating transactions

Picture booking a flight, then a hotel, then a rental car, all as one trip. If the car booking fails at the end, there's no magic "undo the whole trip" button. You cancel the hotel and refund the flight yourself, one step at a time, in reverse order. A saga replaces one single atomic transaction with exactly this idea: a sequence of separate local transactions, each paired with its own compensating action that undoes its effect in spirit, if not literally. There's no global lock and no global rollback here, just a forward path and a backward path.

There are two ways to coordinate the steps of a saga:

For anything beyond the simplest case, orchestration, using a durable workflow engine like Temporal, AWS Step Functions, or even your own state-machine table, is the stronger answer. Saying "I'd model this as an explicit state machine with the state saved for every single order" is a strong, concrete thing to say in an interview.

Here's what building a saga actually demands of you.

Compensations undo the meaning of an action, not the action itself. You can't literally un-send an email, but you can send an apology. You can't literally un-charge a card, but you can issue a refund, and the customer's statement will show both entries.

Some steps simply cannot be compensated at all. Order those steps last if you can, or define a pivot point: everything before it can still be undone, but everything after it must be retried until it eventually succeeds, since there's no going back once you pass it.

Every single step, and every compensating action, must itself be safe to repeat, because retries are guaranteed to happen sooner or later.

The in-between states are real and visible to others. Money can already be captured while the shipment is still pending. Give those in-between moments explicit names, like "payment pending" or "paid, waiting on stock," rather than pretending the whole operation happens instantly.

You lose isolation entirely. Some other part of the system can read this half-finished state while it's in progress. The usual countermeasures are a status flag that other operations respect, like status = 'processing', keeping a separate reserved balance apart from the truly available balance, or re-checking everything again right before the final commit step.

Remember: a saga has no rollback. If a later step fails, you run compensating actions for the earlier steps in reverse order; a refund, not an undo.

Idempotency, exactly-once effects, and reconciliation

Distributed systems retry things constantly. That means every effect visible outside the system must be safe to attempt more than once. This is the practical guarantee that actually replaces true atomicity here.

The mechanisms for this, from weakest to strongest:

1. Operations that are naturally safe to repeat. Setting status='shipped' directly is safe to repeat; incrementing a counter with something like status = next(status) is not. Setting an absolute value beats adding or subtracting a delta.
2. Conditional writes. A statement like UPDATE orders SET status='paid' WHERE id=? AND status='pending' has no effect if run a second time, and the number of rows it actually changed tells you whether this particular attempt was the one that won.
3. Idempotency keys backed by a unique constraint, covered in the earlier API lesson: the general-purpose mechanism for answering "have I already done this?"
4. A ledger, instead of a single mutable value. For anything involving money, never store just a single, changeable balance column. Store immutable rows recording every individual change instead, and calculate the balance by adding them up:

Rows that are only ever added, never changed, give you safety against retries, a complete, auditable trail of everything that happened, and the ability to prove exactly where any number came from. This is genuinely how real payment systems are built, and proposing it in a payments design is a strong signal that you understand the domain.

Reconciliation is the safety net you should always mention, no matter how good your code is. Even with flawless code, distributed systems drift apart over time: a webhook gets missed, a message gets dropped, a compensating action itself fails. Real production systems run a periodic job comparing the two sides against each other, like your own ledger against a payment provider's settlement report, or your inventory count against the warehouse's own count, and either fix small differences automatically or flag them for a person to look at. Saying "I'd add a daily job comparing our ledger to the payment provider's settlement report and alert on any mismatch" is exactly the sentence that shows you've operated a system like this for real.

A few related ideas worth one sentence each. The outbox pattern, the fix for committing to a database and publishing an event as one atomic step, is covered under messaging and queues. Change data capture is how you feed other, derived stores without needing two separate writes at all. TCC, short for Try-Confirm-Cancel, is a variant of a saga where the first step only reserves something rather than committing to it outright; it's the model behind holding a seat, or reserving inventory, for a limited time before it expires.

Remember: never store just a mutable balance for money. Store immutable rows of every change, and compute the balance by summing them; that's what makes retries safe and history provable.

Quick recap