What the interviewer is really checking

Every engineer breaks production eventually, and every engineer makes a call that turns out wrong. Interviewers asking about failure aren't screening for people who've never caused a problem; that person either hasn't shipped much or is lying. They're screening for how you behave once things go wrong, and whether you own mistakes cleanly instead of reaching for an excuse. Your reaction to failure tells them more than your reaction to success ever could.

Three questions that sound alike but aren't

"Tell me about a mistake" can mean three different things, and each one is checking something different:

Picking the wrong shape for the wrong question is a common mistake: an incident story needs mitigate-first instincts, a judgment story needs an honest account of flawed reasoning, and a hard-problem story needs the dead ends you ran into, not just the answer.

Remember: an incident tests pressure, a bad call tests judgment, a hard bug tests persistence. Match the story to the question.

The production incident: mitigate first, diagnose second

The trap to avoid: making a teammate, a tool, or "the deploy pipeline" the subject of the story. Even when a shared system was the proximate cause, talk about your role in the response and the fix, not who's at fault.

Worked example: the Tuesday config change

Situation: "I pushed a config change on a Tuesday afternoon that I believed was a no-op: it was meant to add a new feature flag, defaulted off. Fifteen minutes later we got paged for elevated 500 errors on the checkout API, about 3% of requests. Task: I was the one who shipped the change, so I owned the response. Action: I rolled back the deploy immediately rather than trying to root-cause it live. That's rule one: stop the bleeding before you investigate. Once error rates recovered, I posted a summary in the incident channel with what I'd rolled back and a rough timeline, then dug into logs and found the flag's default value was being read from a config struct with a stale field from an earlier refactor, so "off" was actually evaluating as "on" for about 3% of traffic on a specific code path. Result: I fixed the config bug, added a unit test asserting default flag values match their intended state, and proposed a rule we adopted: any new flag ships behind a canary at 1% of traffic for 30 minutes before a full rollout. We've caught two similar issues at the canary stage since."

Rollback first, root cause second, systemic fix third. That order is what interviewers are listening for.

Remember: stop the bleeding before you investigate. Root cause can wait a few minutes; a growing error rate can't.

The judgment-call mistake: name the flawed reasoning, show the change

The failure needs to be real enough that it stung, but not so large that it makes the interviewer question your judgment overall. A missed estimate, a skipped test, a premature optimization, a feature built without checking whether anyone needed it: these are the right size. Name the mistake plainly. Don't hide it, and don't end with a vague "I learned to be more careful."

Worked example: the custom cache that wasn't worth it

Situation: "Early in a project, I built a custom caching layer for a search endpoint because I assumed Redis's built-in expiry wouldn't give us the invalidation granularity we needed. Task: I was optimizing for search latency, a real problem: P95 was around 800 milliseconds. Action: I spent about a week and a half building a custom cache with dependency tracking, so specific keys could be invalidated when underlying data changed. It worked, but it was complex enough that I was the only one who could safely modify it, and a month later a teammate introduced a bug in it because he didn't understand the invalidation graph. Result: we ripped it out and replaced it with plain Redis expiry plus a slightly more aggressive window, which got 90% of the latency win with a tenth of the complexity. I never validated that the extra 10% was worth the maintenance cost. Since then, I default to the boring solution first and only reach for something custom after I can point to a concrete case the boring solution actually fails on, not a hypothetical one."

The behavior change at the end, defaulting to boring and proving the need before building custom, is a checkable habit, not a platitude.

Remember: end a mistake story with a specific habit you changed, not "I learned to be more careful."

The technical obstacle: keep the dead ends in

Most candidates under-explain this section by skipping straight to the answer. The wrong theories are what make the story credible: a problem where your first guess is correct doesn't read as hard.

Worked example: the sharding bug that took two days

Situation: "I was migrating a service from a single Postgres instance to a sharded setup, and after cutover, a specific class of queries that joined across two tables started timing out intermittently, about 1 in 200 requests. Task: I owned the migration, and it was blocking the rest of the team from shipping features against the new schema. Action: my first theory was connection pool exhaustion, so I bumped the pool size. No change. Then I suspected lock contention and added query logging. Still nothing obvious. It took two days of adding timing instrumentation around every step before I found it: the shard-routing layer was occasionally computing the wrong shard key for a join when a row had recently moved during a rebalance, causing a cross-shard fallback down a much slower query path. Result: I fixed the routing layer to check for in-flight rebalances before computing shard keys, added an alert on the cross-shard fallback rate, and documented the rebalance race condition for anyone building sharded features on that system. It's the reason I now always add fallback-path monitoring for any routing logic, even when the fallback "shouldn't" trigger often."

Common mistakes

• Blaming a tool, a teammate, or the deploy pipeline instead of describing your own role in the response.
• Skipping straight to the fix in a hard-problem story and leaving out the dead ends.
• Picking a "mistake" so small it reads as a humblebrag, or so large it raises doubts about your judgment overall.
• Ending any of these stories with a vague lesson instead of a concrete, checkable change.

Remember: the systemic fix at the end of an incident story matters more than the fix itself. It's the proof this class of bug is now harder to repeat.