Your Eval Set Is Lying to You: Data Leakage in Small Fine-Tunes

My fine tuned ticket triage model scored 88.7 percent on held out data, and I want to talk about how the honest number was 6 points lower, because the way I found out is the useful part.

The eval set was a random 10 percent split of the labeled data. Random splits feel rigorous, and for small real world datasets they are a leakage machine, because real data is full of near duplicates. Support tickets especially. The same outage generates 30 nearly identical tickets, a random split puts 27 in training and 3 in eval, and the model scores on those 3 by memory, not skill. That is not generalization being measured, it is recall being laundered into accuracy.

So I wrote the check I should have started with:

Fourteen exact duplicates across splits, sixty one near duplicates by minhash, and, the quiet one, half the eval set was dated before the newest training examples. Time leakage matters because the deployed model will only ever see the future, and evaluating it on the past, interleaved with training data from that same past, imports knowledge of vocabulary shifts, new products, and new failure modes it will not have in real life.

The clean protocol for small fine tunes, learned the humbling way. Split by time, train on everything before a cutoff, evaluate strictly after. Deduplicate across the boundary, near duplicates included, and cluster by incident or thread so siblings stay on one side. Then expect the score to drop, and treat the drop as good news, because that is the gap between the number you had and the number that was true.

The model was still worth shipping at 82. But I have made real decisions on fake deltas smaller than 6 points, and so have you.