Distributed Systems Basics
The moment your backend runs on more than one machine, you inherit a new class of failure: what happens when the network drops a packet, or two machines both think they're in charge? This lesson covers the CAP theorem, retrying safely, and the idempotency pattern that makes retries safe in the first place.
CAP theorem: the choice you actually have to make
CAP says a distributed system can guarantee only two of three properties during a network partition, a moment when some nodes can't talk to others:
- Consistency: every read sees the most recent write, or an error.
- Availability: every request gets a real, non-error response.
- Partition tolerance: the system keeps working despite dropped or delayed messages between nodes.
Here's the catch: partitions will happen on any real network, so partition tolerance isn't really optional. The actual choice is CP vs AP, and only during a partition. A CP system (a Postgres primary with synchronous replication, etcd, ZooKeeper) refuses a write it can't confirm is durable across nodes, giving up availability. An AP system (Cassandra in its default configuration, DynamoDB) keeps answering requests on both sides of the partition and reconciles conflicts afterward, giving up strict consistency.
The right question isn't "which system is better," it's which failure mode actually fits the feature. A payments ledger wants CP: better to reject a write than record two different balances. A social media like-counter wants AP: better to show a slightly stale count than go down entirely.
Remember: partition tolerance isn't a real choice, it's a fact of networks. The actual decision is CP vs AP, and it should follow from what the feature can tolerate, not a general preference.
Eventual consistency and read-your-own-writes
Eventual consistency is a guarantee that if no new writes happen, every replica eventually converges to the same value, with no fixed bound on how long that takes, though in practice it's usually milliseconds to seconds. It's what AP systems offer instead of strong consistency. Concretely: you write to replica A, a read hits replica B a moment later and gets the old value, and a read shortly after that gets the new one once replication catches up.
The common complaint about this in interviews is read-your-own-writes: a user posts a comment, refreshes, and doesn't see it, because their read landed on a lagging replica. Standard fixes: route a user's reads to the primary (or the replica that served their last write) for a short window afterward, or use a session token that pins reads to a replica at least as fresh as that user's last write.
Remember: eventual consistency means replicas converge, eventually, with no fixed deadline. Read-your-own-writes is the specific, user-visible bug this causes if you don't route a user's own reads carefully.
Retry with exponential backoff and full jitter
Retrying immediately on a struggling downstream service makes things worse: a thundering herd of retries piles onto a service that's already struggling. Exponential backoff spaces retries out; jitter stops many clients from retrying in lockstep.
import random, time
def retry_with_backoff(fn, max_attempts=5, base_delay=0.5, max_delay=30.0):
for attempt in range(max_attempts):
try:
return fn()
except (ConnectionError, TimeoutError):
if attempt == max_attempts - 1:
raise
delay = min(max_delay, base_delay * (2 ** attempt))
delay = random.uniform(0, delay) # full jitter
time.sleep(delay)"Full jitter," picking a random delay between 0 and the computed maximum rather than a fixed exponential value, is the detail that separates a correct answer from a memorized one. It decorrelates retry timing across many clients hitting the same failing service at once, which is why it beats no-jitter and "equal jitter" backoff under real contention.
Remember: full jitter picks a random delay up to the backoff ceiling, not the ceiling itself. That randomness is what stops many clients from retrying in lockstep.
Idempotency: making retries safe to send twice
"How do you handle a network partition?" for a write, the honest answer is you can't always tell whether your write succeeded or just the response got lost. So the client retries, and the server has to make that retry safe. That's idempotency: applying the same request N times has the same effect as applying it once.
@transaction.atomic
def create_payment(request):
body = json.loads(request.body)
idempotency_key = request.headers.get("Idempotency-Key")
if not idempotency_key:
return JsonResponse({"error": "Idempotency-Key header required"}, status=400)
existing = IdempotencyRecord.objects.select_for_update().filter(key=idempotency_key).first()
if existing:
return JsonResponse(existing.response_body, status=existing.response_status) # don't re-charge
payment = Payment.objects.create(amount=body["amount"], account_id=body["account_id"])
response_body = {"payment_id": payment.id, "status": "created"}
IdempotencyRecord.objects.create(key=idempotency_key, response_status=201, response_body=response_body)
return JsonResponse(response_body, status=201)The idempotency key is generated once, client-side, before the first attempt, and sent on every retry of that same logical request. The lookup-and-insert happens inside the same transaction as the actual write, so a crash between "create the payment" and "record the key" is impossible: either both commit or neither does. POST endpoints that mutate money, inventory, or anything non-repeatable should support this header. PUT/DELETE are naturally idempotent by HTTP's own rules, though a PUT is only idempotent if it's a full replace, not a partial increment.
Remember: the idempotency key is generated once by the client, before the first attempt, and reused on every retry. Generating a new key per retry defeats the entire mechanism.