Issue #2: The Pattern I Use for Every Slow Dependency, Not Just LLMs

Ready to transform artificial intelligence from a buzzword into your personal revenue generator?

HubSpot’s groundbreaking guide "200+ AI-Powered Income Ideas" is your gateway to financial innovation in the digital age.

Inside you'll discover:

Download your guide today and unlock a future where artificial intelligence powers your success. Your next income stream is waiting.

Get Your Guide

Hey — welcome back to The 2AM Postmortem.

If you're new here: I'm Vrinda, an SDE2 at Salesforce, previously at Amazon and Walmart. Every week, one real backend incident or design decision, no textbook theory.

This week's Medium piece is about the time we bolted an LLM call onto a save action, it worked great in the demo, and three weeks later our support queue filled up with one word: "slow." The model call sat directly in the critical path — meaning its latency variance, timeouts, and retries all became our save button's problem, not a separate concern off to the side.

Read the full incident on Medium →

The article covers what we changed and the questions worth asking before shipping any LLM feature. What I want to add here is the part that generalizes past AI entirely — because this wasn't really an LLM problem. It was a "slow third-party dependency in a critical path" problem that happened to be wearing an LLM costume.

Once you strip away the AI framing, this is the exact same shape as: a payment gateway call, a third-party address-verification API, an email-sending service, a fraud-check call — anything where your system's response depends on someone else's server having a good day.

The pattern that fixed it, and that I now apply by default to any new dependency like this:

1. Ask "does this need to block the response, or can it happen after?" before writing a single line of integration code. Most of the time, the honest answer is "after." The instinct to make it synchronous almost always comes from how easy it is to prototype, not from an actual product requirement.

2. If it must be synchronous, give it its own timeout — never inherit the parent request's timeout. A slow dependency should fail itself on a clock you set deliberately, not silently eat into the time budget of the whole request it's embedded in.

3. Decide your degraded state before you need it, not during an incident. "Summary unavailable" is a decision. A blank screen or a failed save is not a decision — it's what happens when nobody made one. Write down, for every external call you add, what the user sees when that specific call fails.

4. Treat retries as a cost decision, not a default. Before wiring up automatic retries on anything slow or flaky, ask what a retry actually costs — in money, in load on the dependency, in latency the user feels. Retries silently multiply all three, and "the client library retries by default" is not the same as your team deciding that's the right tradeoff.

This four-question pattern is now just part of how I evaluate any new external call before it ships — not just AI features. If you're integrating anything you don't control the uptime of, run it through these four before you write the integration code, not after it pages someone.

Before you go:

If you're prepping for system design interviews, I put together a free cheat sheet — the questions I actually ask candidates, and what separates a pass answer from a fail one. Download it here.

Got a slow-dependency story of your own? Hit reply — I read everything.

Talk soon, Vrinda