Two Samples Disagreed, So We Changed Nothing

Clad turns a photo and three questions into a personal colour card. Underneath, the verdict is not one model's opinion. Three signals are fused under fixed weights: a pixel measurement taken from the photo, the quiz answers, and a vision model. Each signal can be scored on its own against a label, which is the only reason the story below was possible.

In early September I re-measured a set of production results that carried a label — the offline diagnosis the person remembered receiving. On one axis, light versus dark, the model scored below chance. The signal holding the largest weight was the one getting it wrong most often, and the fused result came out worse than the pixel measurement would have on its own. The model was not helping. It was pulling the answer down.

The fix was finished before the decision was

There was a precedent, which made it worse. Months earlier the warm-versus-cool axis had shown the same shape, and the answer then had been to set the model's weight on that axis to zero. That decision had held up. The light-dark weight had never been through the same check, and here it was failing it.

So everything was ready. The weight would go to zero, the pixel constant would be recalibrated against the production labels, the tests would be updated. I had the numbers, the precedent, and a change that took an afternoon. What I did not have was a reason to doubt it — which is exactly the state in which a product's judgement gets changed on the strength of one sample.

Before merging, I ran the same computation on the other set I had: a golden set of a few dozen people with labels assigned by an expert rather than remembered by the subject.

The second sample pointed the other way

It came out reversed. On the golden set the model was the better signal and the pixel measurement the worse one, and the recalibrated pixel constant — the one production data had asked for — scored worse still. Two samples, both small, each pointing firmly in the opposite direction, and the gap between them not large enough to call significant.

Each sample had its own tilt, and once I looked for it the tilt was obvious. The golden set is built from press photographs: even light, neutral backgrounds, faces that were made up for a camera. Pixel measurement is at a disadvantage there. The production labels are what people remember being told in an offline session, possibly years ago, possibly by someone using a different system. They wobble in their own way. Neither set is wrong. Neither is clean enough to overrule the other.

That left one honest conclusion: the data cannot decide this. So the weights stayed where they were, the constant stayed where it was, and the change that had been ready for two days was closed without merging.

What both samples said survived

Changing nothing was not the same as learning nothing. When two samples disagree about almost everything, the small thing they agree on becomes more valuable, not less, because it has survived two different biases.

Both sets agreed on this: on the light-versus-deep axis, the product's verdict is close to a coin flip. Not on which signal is at fault — they disagreed about that — but on the fact that the axis itself is where results wobble. That is a narrower finding than "the model is broken", and it turned out to be the more useful one. It pointed at a different fix entirely: not raising accuracy on that axis, but telling the person when their result sits on its boundary. The product already computed that. It had never said it. That change shipped instead, and it is a separate note.

A second sample that can be wrong

The pattern I want to keep is this. When measurement points strongly in one direction and a precedent lines up behind it, the fix gets finished before the judgement does. At that point what you need is not confirmation — you already have that — but a second sample that is allowed to disagree. Not a re-run of the same data. A different set, gathered a different way, with its own tilt.

If it agrees, you have lost an afternoon and gained a change you can defend. If it disagrees, the conclusion is "neither", and that is not a failure. It means the product's verdict was not rewritten on the strength of one sample. The temptation is to read "neither" as indecision. It is the opposite: it is the one decision that both samples actually support.

And whatever the two samples still agree on after the disagreement is where the next change should start. In Clad's case it was smaller than the change I had prepared, cheaper to ship, and more useful to the person holding the result.