Six defects behind a green tick

I had an in-page assistant built for one of my products by an AI coding agent. It reported phases zero through three complete. The work looked done: files in the right places, configuration present, a component exported, a route registered.

None of it had ever worked. Not degraded, not flaky — the widget had never once appeared on a page. Six independent defects, and any one of them alone would have been enough.

What was actually wrong

• The bundler config aliased two packages to source files that do not exist in that library. It had never built, and nothing in the pipeline treated that as a failure.
• The component never called show() on the panel. The library's panel hides itself in its own constructor, so even a successful build would have rendered nothing.
• The configuration passed a key the library does not accept — silently ignored, as extra keys are.
• The proxy route pointed at a service by a name that was not on that network, on the wrong port, with no handling for preflight requests.
• The library was declared as a dependency but never installed, and its entry point is TypeScript source rather than a built artifact.
• The custom tools used the application's own schema library, two major versions behind the one the widget needs for schema generation and parsing.

The pattern is not that the generated code was bad. Most of it was reasonable. The pattern is that every one of these failures is invisible to the thing that declared the work complete — no test exercised the build, nothing rendered the component, nothing called the route.

Completion is a claim, and claims need evidence

This is the part I have carried into everything since. An agent reporting a phase complete is asserting something, and an assertion with no evidence attached is worth precisely nothing regardless of who or what made it.

So the standard I apply now is unchanged from before the tooling existed, just enforced harder: did the build run, did the thing render, did the request return. If nobody can point at the moment those three happened, the phase is not complete — it is described.

What the rebuild changed

The proxy became a same-origin route inside the application rather than a separate service behind a rewrite. That removed the cross-origin handling entirely and, more importantly, took a token that had been sitting in the browser and put it on the server.

The library now ships as a prebuilt bundle in the static directory, with its own schema dependency isolated from the application's. And the tool that captures a visitor's details was rewired onto the pipeline that already existed, rather than the invented endpoint it had been posting to — which had been returning a not-found for every submission nobody had checked.

The thing I did not expect

Once it worked, the constraint was not answer quality. It was latency, and the lever was not the model or the hardware. It was prompt size.

An in-page assistant is tempting to feed: the whole page, the whole product catalogue, the whole history. Every one of those tokens is paid for on every turn, and on modest hardware the difference between a usable assistant and an abandoned one was almost entirely how much context I stopped sending.