The Fifth Surface: The Parity Check That Enumerated Its Own Blind Spot

My site publishes essays in more than one place, so a while ago I wrote a test to stop them drifting apart. It compares the essay list rendered on the home page, the static copy of that list in the HTML, sitemap.xml, and llms.txt. All four have to agree, exactly, or the build fails.

It has been green every night since I wrote it. Last night it passed 371 assertions and reported zero failures.

Tonight I opened 404.html for an unrelated reason and counted fourteen essays on a site that publishes fifteen.

The test was correct and the site was wrong

Nothing was broken in the test. It asserted precisely what I wrote it to assert: those four things agree. They did. They still do. The 404 page also lists every essay, hardcoded in the same way the static fallback is, and it was never one of the four — so when I shipped the fifteenth essay and dutifully updated home, sitemap and llms.txt, the suite confirmed I'd done a complete job of a task I'd defined incompletely.

A 404 page is a low-stakes place to be wrong, which is exactly why it's a good demonstration. Nobody noticed for a day. If the same shape of mistake had been in a pricing table or a permissions matrix, nobody would have noticed for a day either.

An invariant that enumerates its own members is only ever as current as the last person who edited the list.

Why "just derive it from disk" is the wrong fix

The obvious correction is to stop naming surfaces and start deriving them: every essay file in the directory must appear everywhere. I nearly did that, and it would have been wrong, because there are twenty-one essay files on disk and only fifteen are published. Six are deliberately unlisted — an older run of writing in a voice I've since moved away from, kept at their URLs so existing links don't rot, marked noindex, and intentionally absent from every index.

So the naive derivation fails in the other direction: it turns a deliberate decision into six loud failures every single night, and a test that cries wolf nightly gets muted within a week. That's a worse outcome than the bug, because now you've lost the check and the bug.

The distinction that matters is which half you derive. The set of essays is a decision — it belongs in one canonical file that a human edits on purpose. The set of places that list essays is a fact about the repository, and facts should be discovered, not typed. My check had it backwards: it hardcoded the surfaces and trusted a human to keep the content aligned, when the human is the unreliable part.

What I changed

The parity check now scans every shipped HTML file for links matching the essay URL pattern. Any file containing more than a couple of them is treated as an index — a surface that claims to enumerate the writing — and is asserted against sitemap.xml, which stays the single canonical list. New surface, automatically covered. No registration step, because a registration step is the thing that failed.

Then I did the part people skip: I reintroduced the defect and watched the test go red. I deleted an entry from the 404 list, ran the suite, confirmed the failure named the file and the missing slug, and put it back. An assertion you've never seen fail is a hypothesis, not a test — that's the same lesson I keep relearning about generated tests, and it applies just as well to ones I wrote myself at midnight.

The pattern, for anyone with a bigger blast radius than a 404 page

This is the third time in a month I've written some version of the same sentence, so I'll state it plainly: my tests keep failing at the edge of their own enumeration. A few weeks ago it was a test that blocked third-party origins when the real failure mode was origins that stall instead of failing. Now it's a test that named four surfaces when the repo had five. Different bugs, one shape — I wrote down the instances I could think of and the suite faithfully guarded exactly those.

Three questions I now ask of any invariant, on my own site and in review at work:

Does this assertion contain a list I typed? If yes, the list is a maintenance obligation nobody scheduled, and it will be wrong within two quarters. See whether it can be discovered from the filesystem, the schema, the route table — whatever the source of truth actually is.

What is the canonical source, and is everything else downstream of it? "These four agree with each other" is a much weaker claim than "these N agree with the one that decides." Mutual agreement between peers has no direction: four surfaces can be consistently, unanimously stale.

Have I watched it fail? Break it on purpose, once, and read the failure message. If it doesn't name the file and the specific difference, the version of you who trips it at 4pm on a Friday will spend twenty minutes finding what the test already knew.

None of this is sophisticated. It's the difference between a suite that tells you the site is correct and a suite that tells you the four things you thought of are correct — and the second one is much more comfortable to live with, which is the problem.

Green means the things you thought of are fine. It has never meant anything else.