A build service that uses the mirrors

A coordinator holds the queue and runners poll it, rather than the coordinator pushing work out. That one choice is why a build host needs NO inbound ports at all, which in turn is what makes it reasonable to run one on a laptop, behind NAT, or on a machine whose firewall is not yours to change.

Scheduling is by label: a target declares what it requires, and a runner is only leased work whose requirements are a subset of what it advertises. An empty requirement runs anywhere.

Recipes live in the repo as world-build.json, so the build definition is versioned with the thing it builds.

Source is checked out through the git backend and every dependency resolves through the mirrors. So the build is reproducible for the same reason the mirror stays current: both follow from the same routing, and neither needs maintaining separately.

Steps can be cached — the target declares which directories to persist and, optionally, the files whose contents key the cache.

A step can name an image and run inside it, under rootless podman on a runner that advertises container. The workspace is mounted in and the rest of the host is not.

Restoring a tarball of build directories is the easy part and, on its own, buys nothing. A target that declares a cache therefore builds at a stable path rather than in a directory named after the build number, because real toolchains bake absolute paths into what they cache — one of ours restored 1.5 GB by the correct key and compiled from scratch anyway, because two hundred restored files named a workspace that no longer existed.

Its working tree also survives the build now. A re-cloned source file is NEWER than the cached object compiled from it, so a fresh clone invalidates the cache it just restored; the next build fast-forwards the existing tree to the queued commit instead. Correctness comes from resetting every tracked file to that commit, which is a stronger guarantee than deleting the tree and a weaker one than pretending generated state does not exist.

What a reused tree must never do is smuggle something forward. The runner deletes its own per-build files and everything matching the target's declared artifacts before it starts — otherwise a step that silently produced nothing would ship the previous build's output, which is the failure that looks most like success.

A list of builds is ordered by time, so a target that went red three weeks ago has scrolled off the bottom of it — and a target that is broken produces exactly as much output as a healthy one, which is none. So the question is inverted: world targets reports the declared targets whose last build FAILED and the ones that have never run at all.

Never-run is usually the bigger number and wants different treatment from a failure, so a target nobody schedules declares itself on-demand and stops being a finding — while an on-demand target that FAILS is still reported. And a build that never checked anything out says nothing about its target: a bad ref is counted and named, not silently scored as a breakage.

Somebody is told when one STAYS red, and the trigger is a dwell rather than the red/green transition it looks like it should be. The fleet's own history is what refused the obvious answer: transitions arrive in red/green pairs minutes apart, because that is what somebody watching their own push looks like. Of thirty red spells, twenty-five resolved themselves within about an hour and every one that actually needed a person ran past six — so the alarm waits two hours, which sits in the gap rather than in the middle of the data.

Deploying the hub means restarting the mirror every other build is fetching through, so a deploy has to wait for the fleet to be quiet. The obvious implementation is the expensive one: a deploy that is handed to a runner and then blocks is holding a lease the whole time it waits, and the queue stops for work it has nothing to do with. Measured over four days, deploys spent nearly three hours doing exactly that, and other builds queued during those waits started a median of thirteen minutes late.

So a deploy waits before it is leased, where waiting costs nobody anything, and is handed out only once nothing else is in flight. The starvation case — a fleet that is never quiet — is handled by letting a long-waiting deploy reserve the queue, and that reservation expires: a deploy nothing can run must not freeze the fleet forever, which is the failure the simple rule was avoiding in the first place.

An operator can hold the fleet on purpose, too: world drain hold <reason> stops leasing — deploys included — until world drain release. It leaves running builds alone, survives a coordinator restart, needs the deploy token, and every surface that shows the drain names the hold and its reason first. It is a separate record, beside the drain the coordinator works out for itself rather than replacing it.

A deploy also inherits the wait of any deploy it supersedes. Without that, a push arriving more often than the threshold resets the clock every time and the deploy is held off indefinitely — which happened three times in twenty-five minutes before anybody noticed it was structural rather than bad luck.