Getting a model into production is an engineering problem, not a prompt problem.

The default posture is to pick one model provider and send everything to it. That is a procurement decision dressed up as an architecture decision, and it is usually wrong in both directions: you overpay for the easy calls and under-serve the hard ones.

On my own agent platform, draft generation runs on a self-hosted thirty-billion-parameter model, while classification and reasoning still go to a hosted frontier model. Not because local is better — because drafting is high-volume, latency-tolerant and reviewed by a human anyway, and classification is low-volume and unforgiving.

One discipline holds the whole thing together: the split stays provisional until an evaluation justifies moving it. Classification remains on the hosted model for exactly that reason — the measurement that would license the cheaper option does not exist yet, so the cheaper option does not get taken. A routing decision without an eval behind it is a guess, and guesses that save money are the ones that get adopted fastest and reviewed least.

Most teams generate free text, then parse it, then repair it when parsing fails. That is three chances to be wrong. Passing a JSON Schema to the inference layer and constraining generation removes the first two: the model cannot emit a shape that does not fit.

When I first measured this I expected it to be unusable — the number in my head was two to four minutes per call. It was not. Schema-constrained calls came back between one and a half and fourteen seconds. The expensive belief was the remembered number, not the technique.

Constrained does not mean correct. On a six-prompt golden set, parsing succeeded every time once the schema was enforced, but first-pass semantic validity was only half, rising to two thirds after tuning. So the schema catches shape and the validator catches meaning, and both are needed.

In two out of two prose generations, the local model fabricated a statistic. Not a rounding error — an invented figure, stated with the same confidence as the true sentences around it, once with a plausible-looking source attached.

This is the single most expensive failure mode in applied AI, because it is invisible at review time unless the reviewer already knows the answer. It is also why the rule on every system I build is the same: generated prose that asserts a fact never ships without a human who could have written that fact themselves.

The three obvious responses all fail. Review everything does not survive contact with volume. A larger model reduces the rate without changing the kind — and a rarer fabrication is more dangerous, because reviewers relax. A fact-checking pass asks a system with the same weakness to audit itself. What actually works is refusing to automate the category at all: structured output is safe to automate, prose that asserts a fact is not, and treating them as one product is how the failure gets in.

Here is a failure mode worth designing against before it finds you. An agent is marked active and producing content. It is not. A credential is absent, so every piece of output is coming from a deterministic fallback template — real text, plausible text, written by a for-loop rather than a model. In the case I audited: ninety-three drafts.

Nothing alerts, because nothing is broken in the sense a monitor understands. The feature has degraded into a worse version of itself and continues reporting success — which is why this class is found by auditing output provenance rather than by watching dashboards.

The interesting fix is not restoring the credential. It is making the agent emit nothing and log that it cannot run — zero drafts when the model is unavailable, which reads as a regression on a dashboard and is the correct behaviour. The existing ninety-three drafts were then backfilled with their true source and flagged, because a silent degradation leaves a data problem behind it as well as a code one.

I run open-weight models on my own machines, including a thirty-billion-parameter model for text and a local image pipeline. The argument is not cost and it is not ideology. It is that some work should not leave the building, and a client's source code is usually that work.

The trade is real and I will state it before you ask: the box is CPU-only, so latency is the constraint, and prompt size is the lever that moves it more than anything else. A smaller model was not the answer — the four-billion variant returned empty responses because its reasoning phase could not be suppressed, which is the sort of thing you only discover by running it.

Where a hosted frontier model is the right tool, I will say so and you decide. The point is that it should be a decision, with a reason attached.