Watching it after it ships

The evaluation passed, the change went out, and for six weeks nobody looked. Then support mentions that the classifier has been putting billing questions in the technical queue since roughly the start of the month, and nobody can say when it started because nothing was watching.

Nothing broke. There was no error, no alert and no failed request. The output stayed well-formed and became wrong, which is the shape almost every production failure takes in these systems, and it is the shape ordinary monitoring is blind to.

Your error rate stays at zero throughout.

Watch the distribution, not the errors

You cannot label live traffic as it arrives. What you can do is watch the shape of the output and notice when it moves.

For a classifier, that is the proportion in each category. If billing has been 31% of tickets every week for four months and is 12% this week, something changed. It might be your prompt, a model update, a marketing campaign that brought different customers, or a genuine shift in what people are asking. All four are worth knowing about, and all four are invisible in an error count.

The same works elsewhere. Output length. How often a field comes back empty. How often a tool gets called. Rate of "I don't know" responses. Pick two or three that would move if quality fell, chart them by day, and look at the chart.

The point is not to have a number. It is to know what normal looks like, because you cannot recognise abnormal without it.

Sample and grade a few, every week

Distributions catch shifts. They miss a system that is quietly wrong in the same proportion it always was.

So take twenty real outputs a week, at random, and grade them properly. By hand, or with a judge you have validated, or both. Twenty is small enough that somebody will actually do it and large enough to catch a serious regression within a week or two.

Random matters. The temptation is to look at the ones that got complained about, which tells you about complaints. Weight the sample slightly toward the cases you know are hard, but keep most of it random, or you will build a picture of a system that only handles the interesting cases.

Keep the graded samples. In four months they are a labelled test set that came from real traffic, which is worth more than any set you wrote by hand.

Log enough to reconstruct one run

When something is wrong, the question is always the same: what exactly did the model see? Answering that needs the prompt as sent, the retrieved context, the tool calls and their results, the model id, and the output. Miss any one and the investigation stops.

The model id is the one people leave out, and it is the one that matters most when a provider updates the model underneath you. Six weeks later, "it started on the 3rd" is the only thing that separates a prompt change from a model change.

Keep enough of this to look at, be honest about what it contains, and set a retention window on purpose. This is customer data, and a debugging log with no expiry is a liability wearing a helpful hat.

Feedback is a weak signal, so weight it

Thumbs up and down come mostly from people at the extremes, and the rate at which people bother is not the rate at which things are wrong. Treat it as a pointer to cases worth reading, not as a measurement. A run of five thumbs-down on the same kind of question is a strong signal even though the overall percentage is not.

Where this doesn't help

None of this tells you a system is wrong. It tells you something changed. A model that has been misclassifying billing tickets since the day it launched has a perfectly stable distribution, and nothing here catches it. That is what your evaluation set was for, and this does not replace it.

Distributions also need volume. At forty runs a day, weekly proportions swing far enough on their own that you will chase several changes that were noise. Below that, sampling by hand is the only thing that works, and the honest move is to read more of them rather than to build a dashboard that cannot say anything.

And an alert nobody has agreed to act on gets muted within a fortnight. Before you set a threshold, decide who looks and what they do about it.

The move

Pick two output properties that would move if quality fell and chart them daily. Then put twenty random outputs a week in front of a person. Do the sampling first: it is less work than the charts and it catches more.