The 3 A.M. Test: Operating Software the Model Helped Write

The question my team used to ask about a new feature was "can we build this?" For most of my career that was the honest bottleneck — the thing standing between an idea and a shipped product was the weeks of typing. That bottleneck is mostly gone. The model builds it, and it builds it in an afternoon, and it demos clean. So the question stopped being useful, because the answer is now almost always yes. The question that actually predicts whether a feature will hurt us is a different one: when this pages someone at three in the morning, can a tired human who didn't write it figure out what it's doing and make it stop?

That's the 3 A.M. test, and it has almost nothing to do with whether the code is correct. Code review — even good, ownership-focused review — happens when everyone is awake, has context, and can ask the author a question. Operating a system happens when none of that is true. The person holding the pager at 3 a.m. is not the person who prompted the feature into existence, does not have the demo in front of them, and cannot ask the model "why did you shape it this way?" in any form they can trust. What got cheap was writing the code. What did not get cheap — what got quietly more expensive as we generated more — was standing in front of a running system at night and understanding it well enough to intervene.

Generated code is fluent, and fluency reads as safety

The specific trap is that model-written code is unusually pleasant to look at. It's idiomatic, it's commented, the variable names are good. That fluency reads as reliability, and it isn't the same thing. A human writing something risky tends to leave fingerprints — a nervous comment, an over-cautious retry, a "TODO: this is sketchy." The model writes the sketchy path with the same calm confidence it writes the safe one. So the surface no longer tells you where the risk is. We shipped a sync job that looked immaculate and, under a specific retry-storm we'd never demoed, silently doubled a queue every ninety seconds. It was found at 3 a.m. by someone reading logs that didn't explain themselves, because the code that was so easy to generate had never been asked to explain what it was doing while it ran.

Code review asks "do we understand this well enough to merge it?" The 3 A.M. test asks "can we operate it when no one who understands it is awake?" Cheap generation only answered the first question.

What we started requiring before a generated feature ships

We didn't respond by generating less. We responded by treating operability as part of the definition of done, the same way we treat the eval for a model-backed feature. Three things now have to be true before something the model wrote goes to production, and none of them are about correctness. First, it has to say what it's doing: structured logs and a metric at every decision point that would matter during an incident, because a system you can't observe is one you can only restart and pray over. Second, it has to have a blast radius someone chose on purpose — a limit, a timeout, a circuit breaker, a kill switch that a half-asleep person can hit without reading the source. Third, there has to be a runbook a human wrote, in human words, that says what this thing does, how it fails, and what to try first — because the one artifact the model cannot generate for you is the judgment of the person who will be woken up.

That last one is the tell. If nobody on the team can write the runbook, we don't actually understand the feature well enough to run it, no matter how clean the diff looked in review. Writing the runbook is where "the model gave us this" gets exposed — you cannot describe how something fails if you only know that it passed. So the runbook isn't documentation we produce after shipping; it's a gate we pass before it. More than once, sitting down to write "here's how this degrades" is the moment we discovered it didn't degrade at all — it just fell over — and we sent the immaculate, generated code back to grow a fallback.

Why this is a leadership problem, not a tooling one

It would be convenient if the fix were a better linter or an observability vendor, and those help, but the real shift is what a team decides "done" means. When generation was expensive, the cost of shipping something you didn't understand was naturally capped — you had to type it, and typing it forced a slow, grudging comprehension. That accidental safety rail is gone. Nothing stops a team now from shipping a large volume of code that works in the demo and that no living person can operate under load. The only thing that stops it is a standard, held on purpose, that says the pager is a first-class user of every feature we build.

The honest version of "AI made us faster" is that it moved the risk from build time to run time. The demo is no longer where you learn whether something is a good idea; the incident is. So we test for the incident up front. Before we're impressed that the model built something in an afternoon, we ask the only question that survives contact with production: not "does it work?" but "can we run it at 3 a.m.?" If the answer is no, it isn't shipped — it's just a very convincing demo, and we've all learned what those cost.

If you're rethinking what "production-ready" means now that anyone can generate a working draft, I'm happy to compare notes. Reach me at arun@arunjalota.com.