Letting a model do the marking

Grading four thousand outputs by hand is not going to happen, so at some point somebody suggests using a model to grade them. This works, within limits that are narrower and stranger than they look, and the limits are the whole lesson.

Start with the failure. You ask a model "is this a good answer, score it out of ten". You get sevens and eights, for everything, including the answers you know are bad. You conclude the model cannot judge quality. What actually happened is that you asked a question with no right answer and got the average of every answer to that question that has ever been written.

Ask a question with a right answer

The fix is the same one as writing a good exercise. Replace one vague judgement with several specific checks, each of which a careful person would score the same way.

Not "is this summary good". Instead:

Does the summary state a specific renewal date? Answer yes or no, then quote the sentence you based that on.

Now there is something to be wrong about. Two people would agree on the answer, which means you can check whether the model agrees with them, which means you can measure it. Vague criteria cannot be measured, so nobody finds out they are not working.

A weighted rubric of five such checks is worth more than any single quality score, and it tells you which part failed, which is the thing you were going to ask next anyway.

Measure the judge before you trust it

A judge is a model doing a task, so it needs the same evidence as any other. That means labelling by hand, once.

Take fifty outputs. Score them yourself, or better, have two people score them so you know what human disagreement looks like on this task. Now run the judge over the same fifty and compare.

The number that matters is agreement with your labels, and the honest bar is human-to-human agreement on the same set. If two of your own reviewers agree 85% of the time, a judge at 82% is doing the job. Chasing 95% is chasing an agreement level the task does not contain.

Fifty is enough to find a judge that is badly broken. It is not enough to tell 82% from 86%, and anybody quoting a two point improvement on fifty examples is quoting noise.

The biases are specific and known

Longer wins. Given two answers, judges prefer the longer one at a rate that has nothing to do with quality. If your change made outputs longer, expect a score rise for free.

Position matters. In a pairwise comparison the first option gets picked more often. Run every pair both ways round and average. If the verdict flips when you swap them, that pair is a tie and should be recorded as one.

A model flatters its own output. Given a choice between its own answer and another model's, it tends to prefer its own. If you are using a judge to decide whether a prompt change helped, this is survivable because both sides come from the same model. If you are choosing between vendors, it is not.

Scores cluster. Ask for one to ten and you get sevens and eights. Ask for one of four labels with a description each, and the distribution spreads out.

Where this doesn't help

A judge cannot check a fact it does not have. Asking whether a claim about your Q3 revenue is true, with no access to your revenue, produces a confident answer built on nothing. Factual checks need the source in the context, and then you are back to the grounding problem rather than the judging one.

It is also weak on anything genuinely a matter of taste. Whether a headline is funny, whether an email sounds like your company. You can measure agreement on these and you will find it is low, including between people, which is the useful finding: stop trying to automate that part and sample it by hand instead.

And a judge you never re-check drifts away from you. The prompt stays the same while the outputs it grades change shape, and the fifty labelled examples you validated against are two months old. Relabel a fresh handful every so often, or you are trusting a measurement whose last calibration you cannot remember.

The move

Split the judgement into checks a person would agree on, hand-label fifty outputs, and measure the judge against them before you use its numbers to decide anything. Compare its agreement to your own reviewers' agreement, not to 100%.