The Source Is Not the Software

My site has a small deterministic knowledge base behind its "Ask Arun" box. Its most important job is not answering questions about my stack. It's correcting the record: a data broker publishes a specific, wrong employer for my name, and that page outranks mine. So the first branch in the router intercepts every phrasing of "where does Arun Jalota work?" and says, plainly, that the profiles claiming otherwise are a different person.

I wrote that branch. I reviewed it. I can open portfolio.jsx right now and read it, with a comment above it explaining why it must stay first. It has been in the repository for days.

It has also, for those same days, been dead code in the file every visitor actually runs.

The site ships a compiled bundle. portfolio.jsx is the source a human edits; portfolio.js is what the browser downloads. Last night's test run asked the shipped bundle five questions it should have had strong answers to, and got back the wrong answer to all five. "Does Arun Jalota work at Game App Studio?" returned my Unity credentials — which, in context, reads as a yes. So I opened the bundle:

if (false && has("game app studio", "rocketreach", ...))

Two branches, both prefixed with false &&. The identity correction and the one that returns my job title, switched off in the artifact, live in the source. No compiler emits that. Somebody — some earlier automated pass of my own making, editing the generated file instead of the source — had disabled them by hand, and the change had survived because nothing in my pipeline compares those two files.

The check that was watching

I did have a staleness check. It had been passing for months:

mtime(portfolio.js) >= mtime(portfolio.jsx)

The bundle was newer than its source. It was newer because somebody had edited it after the source was last touched. The single fact that should have been the alarm was the fact the check consumed as proof of health.

That's the flaw, and it's not an off-by-one. A timestamp is a proxy for content. It answers "when was this written," and my defect lived entirely in "what was written." The check was never weak at its job; it was structurally incapable of representing the failure. There is no threshold I could have tuned. A file can be arbitrarily newer than its source and arbitrarily wrong.

Nobody reads a generated file. That is the point of generating it, and it is also the whole attack surface.

Every safeguard I have runs on the source. Code review reads the .jsx. Grep finds the .jsx. When I want to know what my site says about my employer, I open the .jsx — and it tells me, correctly, and I believe it, because it's the file I wrote. The .js is 135 KB of output nobody has scrolled through since the day the build script was written. It is the only one of the two that has ever been served to a person.

Why this shape keeps finding me

This is the fifth time this year I've written up the same defect class on this one small site, and at this point the pattern is more interesting than any individual bug.

Each one I fixed by making the check specific. But the row I keep writing is always the same underneath: I asserted on a proxy for the thing, and the proxy stayed true after the thing became false. A header instead of a body. An index instead of an origin. A tag instead of a relationship. A timestamp instead of the bytes.

Proxies are what you reach for when the real measurement is expensive or awkward, and every one of them is honest at the moment you write it. They decouple later, quietly, and the passing test is what keeps you from looking.

What replaced it

The new check compiles the source and diffs it against what ships. Not "is it newer" — is it the same program. If the bundle is anything other than the output of running the build on the current source, it fails, and the message says which file and to re-run the build.

Two details mattered more than the idea.

The test imports the build script's own compile step. My first version pasted the Babel options into the test file. That's a second copy of a config, and it would drift from the real one within a month and start reporting a diff on every clean run — which is how a real check becomes noise and then gets muted. So the build script exports the function that produces the bytes, and both the builder and the test call it. One definition, two callers. If the build options change, the assertion changes with them, because it isn't restating them.

There's a cheap tripwire that runs even without a toolchain. A compiler never emits if (false &&. If that string appears in a shipped bundle, a human edited generated code, full stop — no build environment required to know that. It's a worse check than the diff and it costs nothing, so it runs first and it runs everywhere, including on a machine with no node_modules.

Then I broke it on purpose. I switched off an unrelated branch in the bundle, watched both assertions go red, and put it back. An assertion you have never seen fail is a hypothesis, and I've now been burned enough times by confident green to stop shipping checks I haven't personally watched catch something.

The part I'd tell my team

Generated files are the blind spot in every review process I've worked in, and the reason is social rather than technical. We agree not to read them. That agreement is correct — reviewing compiler output is a waste of a human — but it silently converts the generator into the only thing anyone is actually reviewing. The moment something writes to the output directly, it lands in production having been read by nobody.

So the rule I'd draw is narrow: if a file is generated, the pipeline owes you a check that the file is the generation. Not that it's recent. Not that it's present. That it is byte-for-byte what the generator produces from the current input. Anything short of that and "the source says so" is a statement about a file nobody serves.

My source was right the entire time. It was reviewed, commented, and correct, and I could quote it to you. It just wasn't the software.