All notes

19 September 2026 · 3 min read

Exit code zero is not evidence

A deploy that reported success every time and changed nothing. The signal everyone trusts is the one signal that cannot detect this.

Here is a class of failure no monitoring system will catch for you, because every signal it emits is the signal for success: a deploy that transfers the right files, to a real directory, with an exit code of zero — and changes nothing, because no process on that machine reads that directory.

It defeats a good instinct. Trusting your tools is correct almost always, and the narrow case where it fails is the one where the tool is not lying to you — it is answering a different question than the one you think you asked.

The conventional debugging path runs straight past it. Content not updating points at caching, so you clear caches, force new asset filenames, restart the proxy. Each of those is the correct first move, and each one confirms the artefact on disk is right — which narrows the search to everything except the assumption that is actually wrong.

The move that finds it: interrogate the running system

What breaks the deadlock is to stop reasoning about the system and interrogate the running one. Not the configuration you wrote, not the path in the documentation — the configuration the process currently has loaded:

$ nginx -T | grep -A4 'listen 7001'
listen 7001;
root /root/ltvadx-dashboard/dist;

One command, and the premise collapses. The document root was somewhere else entirely and always had been. Every successful deploy had been writing to a directory nothing reads — which is exactly why every check passed. Nothing was wrong with the deploy. Something was wrong with the question.

Why this class survives

A synchronisation into a directory that exists returns success whether or not that directory means anything. The command is correct, the transfer is correct, the files are correct. Only the destination is wrong — and no stage of the pipeline holds an opinion about the destination, so no stage is able to contradict it.

Success was never evidence. It was the absence of one particular kind of failure, mistaken for the presence of the outcome. Those are different claims, and almost every pipeline conflates them.

The second finding, which mattered more

The abandoned directory was not empty. It held a 222 MB stale build from an older stack, and inside it a world-readable environment file and a service account key. A destructive synchronisation aimed there had always been one command from a security incident rather than a harmless no-op — and because the path was documented as the official target, it would have been run deliberately, by someone following the instructions correctly.

The note now carries the correction, the reason, and the command to re-verify it, so the next person to read it — me, in eight months — does not have to rediscover this.

Manmohan Singh, headshot

Written by Manmohan Singh, who builds the systems that move the money. About.

The build log, by mail

New notes, when I publish them.

One mail per note: what broke, the theory I held before I understood it, and the rule I kept afterwards. Irregular on purpose — I write these when something happens, not to a schedule, and I would rather send you nothing than send you filler.

Your address, nothing else. Not sold, not shared, and never used to sell you anything you did not ask about. Reply “stop” to any of them and I take you off.

If this sounds like your stack, say so.

A teardown is five days and a fixed fee, and you keep the findings whether or not we go further.

Get in touch