Who should label a dataset? Rules, a decision model and two LLMs on the same 300 records

Anything built on public expression data has to decide what each dataset actually contains — patient tumours, a panel of cell lines, mice — and GEO does not record that in a structured field. It is stated, if at all, in free text. We put four instruments on that one question over 300 blinded records: a hand-tuned rule system, a calibrated decision model, and two general language models. The rule system was right 43% of the time. The best instrument was right 69%. And the largest single effect in the study was neither: it was the wording of the question, which moved agreement between two human annotators by 24 points.

The question

Given a GEO series' title, summary, organisms and sample count: what biological material was profiled? Five options, in their final wording:

The label is not academic: where this corpus comes from, material type feeds the dataset ranking directly, so a mislabelled series is one shown to the wrong reader.

The four instruments

• Rules — a regular-expression classifier tuned over months against this corpus, and the one in production. Not a strawman: it knows that resected, cohort, primary human glioblastomas and N tumours mean patient material.
• Decision model — TypeSafe's Jev (jev-latest): state as named fields, a question with fixed options, and back comes the chosen option with a probability for every option. No text is generated.
• Language models — Claude Haiku 4.5 and Claude Sonnet 5, given the decision model's own instructions and criteria as a prompt, asked for a label and a probability.

Identical question text for all four. Anything else would compare prompts rather than instruments.

Sample and labels

The corpus is 26,138 ranked GEO series across 53 cancers. The sample of 300 was stratified over the rule system's label so the rare classes appear at all; because that over-represents exactly the strata where the rules are weakest, every accuracy below is also given reweighted to the corpus's natural composition (patient 28.1%, not stated 30.5%, cell line 26.9%, patient and model 12.9%, animal 1.6%).

Labels were assigned from the record text alone, blind — the annotation file held nothing but title and summary, with every system's answer in a separate key file. Two of the 300 were unresolvable, leaving 298. A second annotator, a molecular biologist, labelled a random 60 independently, twice: once under each version of the option set.

The wording result

The first option set used mixed for "patient material together with a line or xenograft". Agreement between the two annotators was 40.0%, κ = 0.216 — barely above chance.

The disagreements were not about reading the records. Two systematic differences explained most of them: one annotator read mixed as "tumour together with normal tissue" — one of the commonest designs in GEO, for which the option set had no word at all — and cell line as "any material described as cells", including flow-sorted patient cells. The option set was rewritten so each option states what it excludes and both boundary cases are decided explicitly. On the same 60 records:

What is left is genuine ambiguity, along three axes: how much may be inferred when material is implied but not named ("cSCC cells", "glioma cultures"); whether outside knowledge of a famous study may be used when the text is silent; and whether a patient-derived xenograft is patient material, an animal, or both.

Accuracy

Jev and Sonnet 5 are statistically indistinguishable; both sit about 26 points above the rule system. Per class, the difficulty is not where one might expect:

not stated is hard for everything: deciding that a record does not say what it profiled means resisting a plausible guess, and all three models guess. The dominant error for each of them is over-reporting patient and model — 20 to 30 records apiece — on series that name only one kind of material.

Where two people agreed

This is the central result. Where the task is unambiguous to people, every model is right 84–92% of the time; the rule system is wrong on nearly half of even those. Model error concentrates in the ambiguous residue. Rule error does not — it is spread across easy and hard records alike. (The subset is 38 records and skewed toward patient; it is a check, not a benchmark.)

Calibration

Every arm was asked how sure it was. Only some of those numbers behave like probabilities.

Haiku's confidence is close to useless: it claims 1.0 on 73% of the sample and says 0.8 on 63 records it gets right a quarter of the time. Sonnet is better calibrated than we expected — the lowest ECE here — but it concentrates on three values, so a threshold has little to bite on. Jev's spreads across the range and rises with accuracy, with one inversion at 0.6.

The practical question is not accuracy but autonomy: at what stated probability can a label be taken without review, and what is left to read?

Sonnet at p ≥ 0.9 is the strongest operating point in the study: half the corpus labelled autonomously with 13 errors in 298. Jev automates more and accepts more error. Haiku has no usable operating point at all, because its confidence does not separate its right answers from its wrong ones.

Cost and latency

Sonnet costs roughly 130× Jev per record for accuracy that is statistically the same; what it buys is the better high-confidence band. Seven of Sonnet's 300 replies could not be parsed even with a 300-token budget — an earlier 40-token budget truncated 27 of them, a harness error of ours that we corrected and re-ran. A decision model cannot return an unparseable answer, because it returns an option rather than prose.

Limitations

• One task, one domain, 298 records. Nothing here transfers to option sets with many more classes or overlapping definitions.
• The primary annotator was the author, blind to every system's label but not independent of the study. The second annotator labelled 60; κ = 0.493 is moderate, not substantial, and the agreed subset is 38 records.
• The second round reused the same 60 records, so some anchoring to the first round is possible. It was done after the rewrite, without access to the first round's file.
• The rule system is the production classifier of this corpus's own pipeline, tuned by the same author: a strong baseline, but not an independently built one.
• Task-trained classifiers and domain BERT fine-tunes were not re-benchmarked; each needs its own labelled training set, which is the cost that keeps them out of small projects. Given that budget they may match or beat everything here.
• Prices, models and the corpus all move. Figures are from 20–23 September 2026.

What we conclude

1. Report κ or do not report accuracy. The same records, the same annotators and a reworded option set moved agreement 24 points. An accuracy figure without an inter-annotator statistic measures the instrument and the ruler at once.
2. The rule system should be retired from this task. 43% overall, and 53% on records two people agree about, is not a baseline worth patching at the margins.
3. Calibration, not accuracy, decides what can be automated. Two arms tied on accuracy; only one offered a threshold that buys real autonomy, and the cheapest offered none despite claiming certainty on three-quarters of the sample.
4. Write the option set as carefully as the prompt. Most of what looked like model error in the first pass was an option set with no word for the commonest design in the corpus.

Frequently asked questions