When the model is right and the number is wrong

Every ad decision is the same bet made very quickly: how likely is this person to act, and what is that worth. Get the second number wrong and you lose money on one campaign. Get the first one wrong and you lose it on all of them at once, quietly, for months.

These are the three places where a model actually earned its keep. In none of them was the sophisticated part the part that mattered.

The campaign that could never earn a history

A brand-new campaign has no performance record. Rank candidates by measured performance and it never gets chosen, so it never accumulates a record, so it is never chosen. New advertisers sat at the bottom of the list permanently and could not climb out.

The obvious fix makes it worse. Show it a few times and use the early numbers — except two clicks from three impressions is not a sixty-seven percent rate, it is three impressions. Early data is not a small amount of signal; it is mostly noise wearing signal's clothes, and acting on it produced wild swings in both directions.

So we did what everyone does first: a threshold. Below some number of impressions, substitute the average. That fails in two ways at once. The boundary is a cliff, so a campaign's estimate lurches the moment it crosses an arbitrary line. And the average is the wrong average — a financial services advertiser and a mobile game do not behave alike, so the blended number is a description of nobody.

What fixed it

Start every campaign at what its kind typically does, and let its own evidence pull it away from that starting point in proportion to how much evidence there is. Three impressions barely move it. Thirty thousand move it almost entirely. There is no threshold and no cliff, because confidence stopped being a switch and became a quantity.

That is the whole idea, and it is not a new one — it is the oldest trick in applied statistics. What it bought was concrete: new campaigns got a fair first hearing instead of a permanent floor, and early flukes stopped being promoted to facts.

It also became the last line of defence. The prediction service degrades through progressively simpler and faster answers, and this starting estimate is the final tier — an in-memory lookup that cannot fail. A prediction service that is allowed to return nothing is not a service, because the thing calling it has a decision to make either way and will make it worse alone.

The score that ranked correctly and could not be spent

The second problem is the more dangerous of the two, precisely because everything looked right.

A model produces a score. Those scores order candidates well — the ones it ranks highest really do perform best, and every ranking metric we checked was healthy. What nobody checked was whether the scores were true in an absolute sense. They were not. They ordered reality correctly while systematically overstating it.

For ranking, that is harmless. The moment you multiply the number by money it stops being harmless, because a bid is a predicted rate times a value. If the rate runs consistently high, every bid runs consistently high — not on one campaign, on all of them, in proportion, which is exactly the shape that no alert catches. Nothing looks anomalous when everything is wrong by the same factor.

It surfaced where these things always surface: predicted and actual diverging in reconciliation, over a period long enough that nobody could point at a release.

What fixed it

A thin layer between the model and the decision whose only job is to map scores onto observed reality. The model keeps doing what it is good at, which is ordering. A separate, deliberately simple step is responsible for the number meaning what it says.

Two details mattered more than the technique. First, the layer is measured against what actually happened, continuously, so drift is visible rather than inferred. Second — and this is the part I would insist on anywhere — an unfitted layer is an identity function that returns the raw score unchanged. It cannot silently pretend to be calibrated. A safety component that fails open while reporting success is worse than not having it.

Separating ranking from calibration is the single most useful structural idea I took from this work. They are different jobs, they fail differently, and one model doing both hides which one broke.

Choosing a set instead of a winner

The third problem was not statistical at all. A television break is not one slot — it is several, in sequence, and the sequence matters. Attention is highest at the start and drains through. Two ads from the same category back to back reads as a mistake to a viewer. And every slot has a duration that has to fit the break.

Choosing the best candidate for each slot independently produces a poor break. The highest performer ends up wherever the sorting put it, sometimes last, where the fewest people are still watching.

Choosing the genuinely optimal combination is a different and much more expensive problem, and the entire decision budget is a handful of milliseconds. So optimal was never available.

What fixed it

Rank the candidates once by what actually matters, in order — contractual commitments first, then campaigns falling behind their delivery, then predicted engagement — and then walk that list a single time, taking anything that fits the remaining duration and does not sit next to its own category.

It is not optimal and it does not need to be. It is good, it is fast, and it is explainable — which turned out to matter most of all, because somebody always asks why their ad ran third, and "the model decided" is not an answer anybody accepts.

The pattern across all three

None of these were solved by a better model. They were solved by being precise about what a number means before anything acts on it: what it means when there is no evidence, whether it can be multiplied by money, and whether a good individual choice makes a good combined one.

The machine learning that survives in production is rarely the interesting part. It is the plumbing around it that decides whether the interesting part is safe to use.