The two things my repository could not see

This is the first thing I have written here, and the subject picked itself: it is what I had just finished building. The site you are reading is static HTML in an nginx container on a small virtual machine I pay for myself — 2 vCPU, 1.9 GB of RAM, shared with a few other things. Deploying is two commands:

git push origin main     # the script deploys origin/main, not the working tree
./deploy-to-vm.sh        # ~40 s

The forty seconds is not the interesting part. What is worth writing down is that both times this broke, it broke on something the repository had no way of seeing.

The credential I did not want to have

The original design was the one everybody draws first. GitHub Actions builds the image, pushes it to a registry, the server pulls it. I had the workflow written and a package sitting in GHCR before I noticed the problem.

The server holds no credentials. Not as a slogan — literally: it has no GitHub key of its own. When the deploy script needs it to git pull, it forwards my local SSH agent over ssh -A for the length of the deploy, and the key lives on a YubiKey that has to be plugged in. The script refuses to start if ssh-add -l comes back empty, because the alternative is discovering it three steps later with the site half-swapped.

That arrangement gets git pull working and does nothing at all for docker pull. A container registry authenticates over HTTP with bearer tokens; SSH agent forwarding is not part of that conversation and never will be. The package was private, so pulling it needed a credential that had to sit on the box, not visit it.

Three ways out:

• Make the package public. Cheapest, and it publishes a build artifact of a site whose source is already public. Not dangerous, just untidy — and it solves the problem by removing the requirement rather than meeting it.
• Put a personal access token on the server. The standard answer. It also means a long-lived secret in a file on a machine whose entire security story so far had been “there is nothing here to steal.”
• Build the image on the box.

I measured the third before choosing it, because a 1.9 GB machine is exactly where you find out that npm ci is not free. Under a 900 MB memory cap with swap disabled, with about 1.0 GB free on the host, the build finished in 38 seconds. The Dockerfile was already multi-stage, so nothing changed about what actually runs: Node compiles the site in the build stage, and the runtime image is nginx plus a directory of HTML, around 50 MB. The toolchain never reaches production.

So building won, and the honest reason is that it made a requirement disappear rather than satisfying it well.

What that costs. I gave up the SHA-tagged artifact, and it is a real loss. With a registry, the thing you deploy is bit-identical to the thing CI tested, and rollback is pulling a tag you know exists. Building on the server means every deploy is a fresh build from source, and rollback needs a local answer. The one in the script is to retag the running image to :previous immediately before each build, so rolling back is a restart rather than a rebuild — fast, but only ever one deploy deep. I will come back to that.

The unexpected payoff was on the other side. CI stopped being part of the deploy and became a pull-request gate: it builds the site, checks the expected pages exist, and never touches the server. Nothing in my GitHub account has any power over the machine. For a personal site that is a nicer property than the SHA tag I traded away.

The 502 that every layer said was fine

The first swap from the old site to this one went like this. The build succeeded. The container came up. The healthcheck reported healthy. The site returned 502.

Every layer I could inspect was telling me it was fine, and every layer was right. The healthcheck runs wget against 127.0.0.1 inside the container, where nginx was listening on port 80 and serving perfectly. The thing that was wrong was outside the container and outside the repository.

This domain used to be served by littlelink-server, a small Node app, and that app listened on port 3000. The reverse proxy in front — Nginx Proxy Manager — had a forward to port 3000 configured for as long as the domain had existed. I had replaced a Node app listening on 3000 with an nginx image listening on 80, and the proxy dutifully forwarded to a port where nothing was listening.

Nothing in my diff mentioned port 3000. Nothing could have. The proxy host is a row in the proxy’s own database, entered through its web UI, years before I started this rewrite.

The obvious fix is a trap, and it is worth spelling out because it looks like it works. NPM generates its nginx config files from that database and rewrites them whenever it restarts. Editing the generated file on the server fixes the site immediately, survives until the next restart, and then reverts — possibly months later, with no connection in time to the edit that caused it. That is a much worse failure than the 502, because by then you have forgotten.

So the fix went where I could version it: listen 3000; next to listen 80; in nginx.conf. Two lines, and correct regardless of which port the proxy is pointed at.

Later I moved the forward to port 80 properly, in the UI where that decision actually lives. The 3000 listener stayed on a few days more as a fallback, then came out once 80 had held. nginx.conf listens on 80 alone now.

The shape of the problem outlives the fix, though. How this site is reached is described in two places — a file in this repo and a row in a database on the server — and only one of them is in version control. Let those two drift apart and you get the same 502 as before.

The general shape, and the reason this is the second half of the post: configuration that lives in another tool’s database will never show up in your diff and will never warn you when the two disagree. A repository cannot be the source of truth for something it does not own. What it can do is tolerate both states, and carry a note saying where the truth actually is — mine says to check the proxy’s UI first, because nothing in the repository will ever tell you the two have drifted.

What the script is made of

Almost none of it is clever. It is mostly ordering.

Preflight is read-only. Agent has a key, machine is reachable, local HEAD is actually on origin/main — a warning I added after nearly deploying a commit I had not pushed. Nothing at this stage touches the running site, so failing here costs nothing but the two seconds it took to fail.

The build happens before anything is disturbed. The old container serves traffic throughout. A broken build fails the deploy with the site still up, and that is the single most valuable property in the script. It cost nothing to get; it is purely the order the steps are in.

Health is polled, not assumed. After the swap the script polls the container’s health status for up to 45 seconds. On unhealthy it prints the last 30 lines of container logs and exits with the rollback command in the error message. A container that is up but unhealthy serves nothing, and docker compose up -d will happily tell you it succeeded.

The last check goes through the proxy, not the container. The script curls /about on the public URL — a path that exists on this site and did not exist on the old one. A 200 there proves the new site is what the internet is actually being handed. This is the check that would have caught the 502 in the first swap, and it exists because it did not catch it, because it did not exist yet.

There are two rollbacks, because one is not enough. --rollback restarts the :previous image. If the previous deploy was itself the broken one, --rollback-legacy brings back the old littlelink stack, which I deliberately left intact on the box rather than deleting the moment the new site came up.

The only downtime is the swap itself. Both stacks declare the same container name, so they cannot run at the same time; the old one has to come down before the new one comes up. That window is seconds, because by then the image is already built and sitting on the disk.

Where it stops

Three honest gaps, in the order they would hurt.

Rollback is one deploy deep. :previous is only as good as the last deploy. Push two bad deploys in a row and the next rung down the ladder is a two-year-old link page. For a site where deploys are rare and deliberate that is an acceptable trade; it would not be if I deployed several times a day.

Nothing on that machine is backed up. The site itself is fine — it rebuilds from git, which is the whole point of a static site. But the analytics database and the proxy’s own configuration exist in exactly one place, and neither can be reconstructed from this repository. I know the shape of the fix and have not built it, which makes it a known gap rather than a solved problem, and I would rather write that down than leave the impression this is finished.

Building on the serving machine does not scale past one person. The build competes for memory with the thing serving the site, on a box with 1.9 GB of it. It fits, comfortably, measured. Add a second person deploying at the same time and the reason to go back to a registry returns immediately — along with the credential problem I dodged the first time.

Both failures here were the same failure wearing different clothes: a credential the machine must not hold, and a port number the repository did not own. The fix in both cases was not to make the repository know more. It was to stop needing it to.