Hey — welcome to the first issue of The 2AM Postmortem.
If you're new here: I'm Vrinda, an SDE2 at Salesforce, previously at Walmart and Amazon. Every week I write about one real backend incident or design decision — the kind of thing that shows up in system design interviews, except it actually happened. No textbook theory, no "10 tips" listicles.
Quick recap, if you haven't read it yet
This week's Medium piece covers a 2 a.m. incident: a network partition split our data center's regions, and someone in the incident channel said "just fail over reads to the replica, we'll figure out consistency later" — not realizing they'd just made a CAP theorem decision, not an operations one. Nobody had chosen consistency or availability on purpose; the system's default made the choice during the outage, in front of customers.
What I want to do here instead is something the article doesn't have room for: the follow-up questions I'd actually ask you about it in an interview, and why most candidates only get partway there.
The 3 follow-ups I'd ask after you explain CAP theorem
1. "Okay — so which one would you pick for this system, and why?" Almost everyone can define C and A. Very few can commit to a choice for a specific, concrete system without hedging into "it depends." It does depend — but a strong answer still picks a side and defends it with a real constraint (cost of an outage, cost of stale data), instead of listing both options and stopping there.
2. "What does your database actually default to, right now, during a partition?" This is the question that exposes textbook knowledge vs. real experience. Most candidates — and most engineers on the job, honestly — have never actually checked. If you can answer this about a system you've worked on, that's worth more in an interview than reciting the theorem correctly.
3. "Can the choice change per field, or does it have to be system-wide?" This is the one that separates senior answers from mid-level ones. The uniform answer ("we chose availability") is weaker than the granular one ("account balance is consistency-first, display fields are availability-first") — because real systems rarely need the same tradeoff everywhere, and knowing that shows you've actually had to live with the consequences of a system-wide default, not just pick one in theory.
Why this matters beyond interviews
Most people can define CAP theorem. Almost nobody can tell you what their own system actually does right now, mid-partition, without checking. That gap — between "I know the theorem" and "I know my system's actual behavior" — is exactly where 2 a.m. pages come from.
Before you go:
If you're prepping for system design interviews more broadly, I put together a free cheat sheet — the questions I actually ask candidates, and what separates a pass answer from a fail one. Download it here.
If you've had your own CAP-theorem 2 a.m. moment, hit reply and tell me about it. I read everything.
Talk soon, Vrinda