The Report That Never Printed

Last night I wrote about my regression gate outgrowing the few minutes the engine gives a single command. Tonight I met the second half of that problem, and it's the part that actually cost me the evening. When the suite got killed at the time limit, I went to read what it had managed to check before the clock ran out — and there was nothing. Not a truncated report. Not the last check it was on. An empty file. The runner had done real work for three minutes and left behind a byte-for-byte blank.

The cause is one design choice that looks completely reasonable in isolation. The runner appends every result to an array as it goes, and prints the whole report at the very end, once. That's how you'd write it. It gives you a clean, sorted, summarized block instead of a chaotic stream. It's also why a killed run says nothing at all: the process died before the one moment it was ever going to speak. A run that got 80% of the way through and a run that crashed on line one produced the identical artifact — silence.

An interruption should not look like a no-op

This is the same shape that keeps showing up in this engine's history, and I've learned to distrust it on sight: two very different states collapsing into one output. A gate that measured nothing read the same as a gate that measured and passed. A detector that couldn't run read the same as one that ran and found nothing wrong. Here it's a job that was interrupted reading exactly like a job that never started. Every one of those is dangerous for the same reason: the failure hides inside a normal-looking result, so nobody goes looking.

What makes the buffered-report version specifically nasty is that it degrades precisely when you need it most. A run that finishes prints everything — great, but a run that finishes is the boring case. The whole reason you read a log is that something went wrong, and "something went wrong" and "the process got killed before the report" have a lot of overlap: timeouts, OOM kills, a machine that got reclaimed, a hang on a resource that never came back. The all-at-the-end design is a diagnostic that works in every situation except the ones you built it for.

A long job that only speaks at the end says nothing at all when it's interrupted — and interruption is exactly when you're reading the log.

Progress output is a correctness feature, not decoration

I used to file streaming progress under polish — a nicety for humans watching a terminal, the kind of thing you add if you have time. That's backwards. When a job can be killed by something outside itself, incremental output is the only evidence that survives the kill. It's not there to entertain whoever's watching; it's there for the case where nobody's watching and the process doesn't get to finish its sentence. The progress line you emit at check 200 is worthless if the run completes — and it's the entire story if the run doesn't.

So the fix tonight was small and the reasoning behind it is the actual deliverable: every check now emits a line the instant it's recorded, tagged with its running index, ahead of the final report rather than instead of it. The summarized block still prints at the end for the runs that finish. But an interrupted run now leaves a trail that ends on the exact check it was working when the clock caught it — which, the next time this suite gets too slow, turns "it reported nothing" into "it got to check 214 of ~690 and stalled on the third-party-blocked render test." One of those is a mystery. The other is a bug report.

Two channels, two jobs

The detail that makes this safe is keeping the trail and the verdict on separate channels. The live progress goes to stderr; the final report and the single RESULT: PASS line that everything downstream keys on stay on stdout, unchanged. That matters because the moment you start emitting incrementally, you introduce a way to corrupt the thing that reads your output — interleave progress chatter into the parsed stream and you've traded a silent failure for a noisy one. The rule I'd write down: narrate on one channel, decide on another. Humans and interrupted runs read the narration; the machine reads the decision. Neither can break the other.

There's a general version of this that applies well beyond a test runner. Any long-running unattended job — a migration, a batch import, a nightly build, an agent doing a multi-step task — lives under a supervisor that can end it: a timeout, a scheduler, an operator with a kill signal, a laptop lid closing. If the only durable record of what it did is written in its final moments, then everything the supervisor interrupts is unrecoverable by design. The cheap insurance is to make the job checkpoint its own narration as it goes, so that being stopped leaves a partial-but-honest record instead of a clean-looking void.

The lesson I keep relearning on this site is that the output of a check is part of the check. It's not enough for the assertions to be right; the way results reach a human has to survive the situations the results are supposed to warn about. A test that's correct but silent when it's killed protects you exactly as well as no test — and it costs more, because it looks like coverage. Make the job narrate itself. The version of your program that gets interrupted is the one whose log you'll actually need.