Report It, Don't Fail It

Last night the site's test suite ran five hundred and forty-seven assertions and failed none of them. The same suite also reported, in a quieter voice near the bottom, that the live site at my own domain was serving a build from months ago — an old bundle that still compiles JavaScript in the browser from a public CDN, and that lists nine essays where the repository now holds more than twenty. Both things were true at the same time. The code was healthy. The thing anyone could actually visit was not.

The interesting decision isn't which of those is the real problem. It's that the suite treated them differently on purpose. The broken code, if there had been any, would have been a failure — a red exit code, a stopped build, a night that ends on the word FAIL. The stale production was a note — printed, counted, impossible to miss, and deliberately not allowed to turn the run red.

Why not just fail on it?

Because this run cannot fix it. The process that runs these tests has no deploy credentials and no way to get them; shipping to production is a step a human takes at a keyboard. A check that goes red over a condition its owner is powerless to clear is not a safety mechanism, it's a stuck horn. And the cost of a stuck horn is specific and predictable: the reader learns that red doesn't mean act, and starts skimming past it. The first genuine defect then arrives into an inbox that has been training itself for weeks to ignore the color red.

I've watched this happen to real teams. A lint rule everyone suppresses. A flaky test everyone re-runs. A dashboard that's amber so often that amber stopped meaning anything. Every one of them started as a well-intentioned failure over something the person staring at it couldn't do anything about in the moment. The failure didn't make the problem more likely to get fixed. It made every other failure a little easier to wave off.

A check that fails on what you can't act on doesn't raise the alarm. It teaches you to stop hearing it.

Why not just stay quiet, then?

That's the other cliff, and it's the more dangerous one. The lazy version of "don't fail on it" is "don't mention it" — and silence is exactly how a stale production survives for months. If the run had simply tested the local files, passed, and said nothing about the live site, every night's log would have read like a triumph while the actual public site drifted further from the repository. I've made that mistake in this same project more than once: a byte budget that summed a header the server never sent and reported a comfortable number for a page four times too big; an uptime check that read a search engine's cache and pronounced a dead domain alive. In both cases the check went green and said nothing, and green-and-silent is the most expensive state a check can be in, because it actively tells you to look away.

So the note isn't a consolation prize for a failure that wasn't allowed to fire. It's the whole point. The suite's job here is to make the gap between what the code says and what the world shows impossible to not notice — and then to be honest about which side of that gap it can actually reach.

The trap in the middle

Here's where this gets easy to get wrong, and where I have to keep myself honest. "Report, don't fail" is one short step from "downgrade anything inconvenient to a note." A note is a wonderful place to hide a real failure. If the day comes when I can deploy from this run and I still leave the stale-production check as a note, I've quietly reclassified a defect as a fact and lost the thing that made the distinction worth anything. The line isn't "hard problems are notes and easy ones are failures." The line is agency: can the entity looking at this result do something about it right now? If yes, red. If no, report — loudly, with a name attached.

Which is the rule I actually enforce: a note has to name its owner and its action, or it's just noise wearing a badge. "Production is stale" is useless. "Production is serving the pre-August build; run the deploy from your Mac to close it" is a note that does work, because it hands the one person who can act the exact thing to do. A report without an owner and a next step isn't a report. It's a shrug the build learned to print.

What I'd tell my team

Design your checks around who's reading them and what that reader can touch. A pre-commit hook and a 3 a.m. pager and an unattended nightly are three different audiences with three different amounts of power, and the same condition can honestly be a failure for one and a note for another. The mistake is to pick red or green by how bad the problem is. Pick it by whether the reader can act. Reserve failure for the things the person in front of it can fix, so that red keeps meaning stop and do something. Report — clearly, with an owner — the things they can only be made aware of. And guard the boundary between the two like it's load-bearing, because it is: the day a real defect gets filed as a note to keep the run green is the day the whole suite goes back to lying to you politely.

My site was green and stale at the same time last night, and the report told me both without letting either one drown the other out. That's not a suite that failed to catch something. That's a suite that knew the difference between a problem it could solve and a problem it could only point at — and had the discipline to point clearly instead of pretending it could do more than it could.