The Archive That Didn't Exist

This site ships a file called llms.txt. It exists for one reader: an AI assistant that has been asked "who is Arun Jalota?" and wants a canonical, owner-written answer instead of whatever a data broker scraped. I've put real care into it — a stat-led bio, a disambiguation section, the full list of essays, refreshed every night. Near the top of the Writing section it says the essays live at https://www.arunjalota.com/writing/. Tonight I followed that link myself. It was a 404. It had been a 404 since the day I wrote it.

It gets better. The AskArun helper on the home page — the other thing I built specifically so machines and visitors get the right story — closes its "what does Arun work on?" answer with "I write both up at arunjalota.com/writing." Same address. Same nothing behind it. The two surfaces I maintain most deliberately for AI consumers agreed with each other, confidently, about a page I had never built.

Green for months, and correctly so

The nightly gate on this site runs about seven hundred checks, and a decent fraction of them are about links. So how does a dead URL in the two most machine-facing files survive that? Because none of those checks were looking at it, and each of them had a perfectly good reason not to.

The link checker reads HTML and follows every <a href>. No HTML page linked to /writing/. Nobody could click it, so as far as the link checker was concerned it wasn't a link. The parity checker — the one that makes sure every place that lists essays agrees with the sitemap — reads llms.txt, but it does so by extracting essay filenames with a pattern like /writing/something.html. A bare directory URL with no filename matched nothing, so it was neither an essay nor a mistake; it was just prose. And the sitemap parity check compares lists of pages that exist, which is exactly the wrong tool for a page that doesn't.

The address written specifically for machines was the one address no machine ever checked.

I want to be precise about what went wrong, because "add a link check" is not it. Every check did its job. The gap was in what counts as a link. My mental model — and the model encoded in the tooling — was that links are anchor tags in HTML, and text files contain text. That was true when the only reader of a text file was a person, who'd shrug at a wrong URL and click around. It stopped being true the moment I started writing files whose intended reader is a program that will fetch what I tell it to fetch. llms.txt is not a document. It's an index. Every URL in it is a link, whether or not there's an anchor tag around it.

The more you write for machines, the more your prose is a link

This is the part I think generalizes. GEO — making sure AI assistants describe you accurately — is mostly an exercise in writing plain text files that machines will act on: llms.txt, structured data, the stock answers in a helper, a robots file with a sitemap line. All of it is "just text" to a linter and "an instruction" to its actual audience. A model that reads "the essays are at /writing/" doesn't hedge; it either fetches the page or tells the user the page is there. Either way, I've made a claim, and it's a claim that fails silently for the human because the human was never in the loop. Nobody emails you about a 404 that an assistant hit on their behalf. They just get a slightly worse answer and move on.

So the rule I've written down: any own-origin URL in a machine-facing file is a link and must resolve. Not "should probably." Must, enforced. Tonight's gate now walks every URL on my own domain that appears in llms.txt, robots.txt, sitemap.xml, and the compiled helper, and resolves each one to a file on disk — with the specific rule that a directory URL needs an index.html, because "the folder exists" is exactly how this bug hid. It also asserts that it found a plausible number of URLs to check, so an over-narrow pattern can't pass by scanning nothing. That second half is a habit now; a check that can only fail in one direction only finds one kind of problem.

Then build the page it was promising

The other half of the fix was to make the claim true rather than delete it, because the claim was good. An essay archive at /writing/ is something the site should have had: a crawlable, linkable page that lists every published piece, with dates and descriptions and a structured-data list so a search engine or an assistant can cite one essay instead of gesturing at the home page. The reason it didn't exist is that the home page already rendered the list, and I'd never needed a second copy. Which is the same instinct that produced the bug: a page that exists for machines doesn't feel necessary to a person.

It's generated, not typed. The sitemap is the one canonical record of what's published; a small build script reads it, pulls each essay's title, date, and description out of the essay's own markup, and emits the page. A --check mode makes a stale archive a red line, and the gate calls it. I've been burned by hand-maintained lists of essays enough times on this site — a 404 page that sat one essay behind, a JSON-LD block that enumerated 27 when there were 28 — that a sixth copy of the list, maintained by hand, would have been a scheduled defect. The archive is also, deliberately, classified in the test suite as an index surface rather than an essay: it must agree with the sitemap exactly, like the home page and the 404 page do, and it's excused from the per-essay checks that would otherwise demand a byline and an article. That classification is one named file in one place. It looks like a decision, because it is one.

The thing I keep relearning on this site is that "who reads this?" is the question that decides what a check has to cover. For years the honest answer for a text file was "a person, maybe." Now, for some files, it's "a program, definitely, on behalf of a person who will never see the error." Those files deserve the same link discipline as a nav bar — more, really, because the nav bar has a human on the other end who'll notice. Write for machines, and your prose is a link. Check it like one.