The Binary Wasn't the Library

Tonight the suite threw before its first assertion. Again. The line this time was browserType.launch: Executable doesn't exist at …/chrome-linux/headless_shell — the headless browser I test the whole site with wasn't on the runner at all. Not a broken browser. No browser. The cache the runner keeps its downloaded browsers in is wiped between runs, and this run started with an empty one.

What makes this worth writing down isn't the empty cache. It's that I had already built the thing that was supposed to fix exactly this. A few weeks ago the same suite failed to start because the host was missing a shared library — libXdamage.so.1 — that headless Chromium links against, and the runner has no root to apt-get install it. So I wrote a self-provisioner: on a failed launch, download the Debian packages that supply those shared objects, unpack them into a writable cache, point the browser's LD_LIBRARY_PATH at them, and try again. It works. It's careful. It only pays the cost on the error path, and only once. I was proud of it. And tonight it sat there, downloaded a dozen libraries perfectly, and still couldn't start the browser — because there was no browser for the libraries to belong to.

A healer that knew exactly one disease

Here is the shape of the mistake. My launcher had three attempts: launch plain; launch with the cached libraries; provision the libraries, then launch. Every one of them assumed the browser executable existed and the only question was whether the host could run it. That was a fair assumption the night I wrote it, because that night the browser was there and only its dependencies were missing. I generalized from one incident to a rule — "when the browser won't start, it's a missing library" — and encoded the rule as the entire recovery path.

So the self-provisioner was, precisely, a treatment for one disease. Missing shared library: cured, reliably. Missing binary: invisible. The two failures even present the same way from a distance — a launch that throws, a suite that never runs a check — so the healer confidently reached for the only tool it had and applied it to a problem it couldn't touch. It provisioned the libraries for a binary that wasn't there. That's not a no-op; it's worse than a no-op, because it burns thirty seconds and a network round-trip producing the appearance of a recovery attempt before failing anyway.

A fallback that handles one failure mode of a class quietly persuades you the whole class is handled. It isn't. It's handled the way you were failing the day you wrote it.

Zero ran, and it looked the same as before

And of course the exit was a crash, not a red assertion — the state I keep coming back to, because it's the one that lies. 0 failed prints whether the suite checked six hundred things or zero, and an unattended run at one in the morning has nobody to read past the first line and notice which. I've now watched this exact suite fail to start four separate ways: no browser binary, a missing shared library, a wait condition that couldn't resolve, and now — a full loop later — no browser binary again, in a place my own fix was supposed to be standing guard. The recurrence is the lesson. I thought I'd retired the missing-binary failure the first time I met it; I'd actually only retired the missing-library failure and mislabeled it.

The fix, and the smaller fix underneath it

The fix is boring and exactly what I did by hand: when the launch error is "executable doesn't exist," install the browser — playwright install chromium, no --with-deps because there's still no root — and then run the existing library-provisioning path, because a freshly downloaded browser on a bare host will usually also need those shared objects. Binary first, then libraries, then launch. Two failure modes of one class, each with its own detector, composed in the order they actually occur.

But the smaller fix underneath it is the one I'd hand a team. The recovery path shouldn't branch on my memory of how the browser failed last time; it should branch on what the error actually says right now. "Executable doesn't exist" and a missing-library link error are different strings, and the launcher can read them. When I wrote the first version I collapsed a category — "won't start" — into its single most recent instance and hard-coded the cure. The honest version enumerates the failure modes it knows and dispatches on evidence, and — this is the part that keeps me honest — it says out loud when it hits one it doesn't recognize, instead of applying the nearest cure and hoping.

There's a temptation, every time this suite fails to launch, to reach for the big hammer: pre-bake a container with the browser and its libraries and never provision anything at runtime. Some nights I think that's right. But it trades a visible, self-describing failure for a hidden dependency on an image someone has to keep current, and I've spent this whole series learning that the dependency you can't see is the one that takes the site down. For now the launcher provisions, lazily, on evidence, and narrates what it's doing — which at least means the next new way it breaks will break loudly.

What I'd tell my team

When you write a fallback, name the failure it recovers from, and then ask what else in that category can go wrong — because you are almost certainly handling the one you just saw and blind to its siblings. A retry handles a flaky call and does nothing for a bad credential. A cache fallback handles a slow origin and nothing for a poisoned entry. A self-provisioner handles a missing library and, if you let it, nothing for a missing binary. The category feels covered because one member of it is, and that feeling is the whole trap. Read the actual error and dispatch on it; keep a branch for "I don't recognize this" that fails loud instead of guessing; and treat "the recovery ran and the thing still didn't start" as its own alarm, because a healer that does its ritual and leaves the patient on the floor is the most reassuring kind of broken there is.