The Test Suite Has a Deadline

The regression suite that guards this site does real work: a headless browser walks every page at two viewports, checks structured data, blocks third-party origins to prove the text still paints, resolves every internal link, and reconciles the essay list across every surface that publishes one. Hundreds of assertions. Tonight it ran into something it had never hit before — not a failing check, but a clock. The environment the engine runs in gives a single command a hard ceiling of a few minutes, and the full suite, grown one honest check at a time, no longer fits inside it. It didn't go red. It got killed partway through and reported nothing at all.

The reflex fix is one line: raise the timeout. I did the equivalent to get tonight's run green. But the reflex is treating a symptom, and the symptom is pointing at something I'd never named. A test suite is code with a deadline, and I had been budgeting everything about it except the one property that had just failed — how long it takes to answer.

A gate that doesn't finish isn't a gate

We talk about tests failing as if red is the only bad outcome. But a gate has three states, not two: pass, fail, and didn't finish. The third is the dangerous one, because of how it tends to get resolved. When a suite runs past its window, nobody sits with the discomfort for long. Someone wraps it in a longer timeout, or splits it so only part runs on each change, or quietly moves it from "blocks the merge" to "runs nightly, we'll check the email." Every one of those is a reasonable-sounding decision that ends the same way: the check still exists, still looks green in the places people glance at, and no longer actually stops anything.

That's worse than a flaky test, and it's worth being precise about why. A flaky test is loud — it goes red when it shouldn't, it annoys people, it demands attention even when the attention it demands is irritated. A suite that's too slow to finish fails in the one direction nobody notices: it stops looking, and not-looking produces the exact same silence as looking-and-finding-nothing. I've written before about a detector that couldn't run for two weeks and how its silence read as health. A suite that runs but can't finish inside its window is the same failure with better optics: it's not obviously broken, it's just gradually, respectably, ceasing to protect you.

A suite's runtime is not a developer-comfort issue. It's a correctness property: the check that finishes is worth more than the thorough one that gets skipped.

Runtime is a budget, like bytes

This site already enforces a budget I'm proud of: the home page may not ship more than 700 KB, and — after a bug where the guard summed to zero and passed on a 1.2 MB page — it also fails if the total comes in implausibly small, because a check that can only fail in one direction can only find one kind of problem. I had a ceiling on the bytes a visitor downloads and no ceiling at all on the seconds my own gate consumes. The asymmetry is silly once you see it. Both are resources. Both drift upward the same way — one honest addition at a time, each individually justified, none of them the moment anyone would call a regression. Payload creep and runtime creep are the same disease; I had a thermometer for one of them.

So runtime gets a budget now, and the budget gets printed. Every report states how long the suite took and how close that is to the ceiling, the same way it already states verification age. A number you look at every night is a number whose drift you notice; a number nobody records is free to double before anyone asks. The point of writing it down isn't the writing — it's that "the tests are getting slow" becomes an observation with a date on it instead of a feeling someone eventually mentions in a retro.

What a slow suite does to a team

On a personal site the cost of a bloated gate is that a nightly job gets truncated. On a team it's more expensive, because a slow suite doesn't just wait — it changes behavior, and the behavior it changes is exactly the behavior the suite exists to encourage. A pipeline that takes forty minutes teaches people to batch unrelated changes into one big merge so they only pay the tax once, which makes every failure harder to bisect. It teaches them to skip the local run and let CI find out, which moves discovery later and makes it cost more. It teaches them to merge on amber — "the slow integration suite is probably fine, I'll watch it" — which is the moment the gate stopped being a gate and became a suggestion.

None of those are discipline failures. They're rational responses to a tax, and if you want the behavior back you lower the tax rather than lecture about it. Which is why "how long does CI take" is a reliability question a technical lead should be able to answer with a number, not a vibe. The suite's speed is part of its correctness, because a correct check that people route around protects nothing.

The fixes that aren't "raise the timeout"

Once runtime is a budget you approach it the way you approach any budget you're about to blow. You tier: a fast smoke gate that runs on every single change and answers in seconds — does every page still return content, is the JSON-LD valid, do the internal links resolve — plus the full browser-level suite on a cadence that can afford it. The mistake is running one monolithic gate and tuning its frequency; the move is running two gates at two speeds and being honest about which one blocks what. You parallelize the work that's independent — my two viewports were walking the site in sequence for no reason other than that's how the loop was written. You delete, which is the one nobody does: a check that has never failed and no longer corresponds to a way the system can actually break is not free insurance, it's rent, and it's paid in the seconds that eventually get the whole suite skipped. And you make the budget itself a check, so the suite fails loudly the day it grows past its window instead of getting quietly wrapped in a bigger number at 1 a.m. by whoever's on call — which, tonight, was me.

The check that matters is the one that finishes. A thorough suite you can't afford to run is a worse guardrail than a lean one you run on every change, because the thorough one's protection is theoretical and the lean one's is real. I spent a long time optimizing what my tests assert and no time at all on whether they'd answer before the clock ran out. Budget the time like you budget the bytes — with a ceiling, a number in the report, and the willingness to throw away the checks that stopped earning their seconds.