Using Jev, a decision model, to grade biological evidence at scale

GSE147766 is titled “The immune cell atlas of human neuroblastoma”. Our regex saw the word cell and filed it as a cell-line study, which pushed it down the dataset ranking on the neuroblastoma briefing. It is nothing of the sort: it is single-cell sequencing of tumours from children, exactly the kind of patient data a researcher opens that page to find. A decision model read the same title and summary and said patient, with probability 1.00. This post is about what happened when we let it look at everything else.

What the briefings are, and where the regex sits

BioTransfer publishes 53 cancer briefings, rebuilt on a schedule from public APIs: which genes are being targeted, which drugs carry an FDA label naming the disease, which trials are open, which GEO datasets are worth opening first, where the funding and the mutations sit. Every figure carries its source and retrieval date. None of the prose is written by a language model.

Between the raw sources and the page there are three classification steps, each of which used to be a regular expression:

The regexes are not naive. The polarity rules know that “Limitations of Use” opens a negative clause, that a negative only governs if it comes before the disease name or within fifteen characters after it, and that a bullet under “indicated for” inherits the heading's polarity. They were tuned by hand against hundreds of labels, and they catch the cases that matter most: a drug whose label says efficacy has not been demonstrated in this disease is a rejection, not an approval, and the page shows it as one. But a regex is a pattern over characters, and a title like “Neuroblastoma patient-derived xenograft” contains the word patient.

What a decision model is

Jev, from TypeSafe, is not a chat model. You send it a state — the evidence, as named fields — and one or more questions, each with a fixed list of options and a line of criteria per option. It returns the chosen option and a probability for every option. It does not write anything. It cannot answer outside the options, so there is no output to parse and no retry loop for malformed JSON.

POST https://api.typesafe.ai/v1/systemone
{
  "state": {
    "title":     "The immune cell atlas of human neuroblastoma",
    "summary":   "Our current knowledge of the different immune cells in neuroblastoma is based on in vitro and in vivo studies mainly focusing on a single cell type ...",
    "organisms": "Homo sapiens",
    "samples":   "19"
  },
  "questions": {
    "cohort": {
      "type": "choice",
      "instructions": "A public GEO expression series is described by its title,
                       summary and organisms. What biological material was profiled?",
      "criteria": {
        "patient":     {"means": "primary tumours, biopsies, resected specimens or a patient cohort"},
        "cell line":   {"means": "established cell lines only, in vitro"},
        "xenograft":   {"means": "PDX, cell-line xenografts or a mouse model only"},
        "mixed":       {"means": "patient material together with cell lines or xenografts"},
        "unspecified": {"means": "cannot be told from the text"}
      }
    }
  }
}

→ {"cohort": {"choice": "patient",
              "probabilities": {"patient": 1.0, "mixed": 0.0, "cell line": 0.0, ...},
              "confidence": 1.0}}

The part that matters for a pipeline is the last line. The probability is calibrated: it is meant to be the actual frequency with which that answer is right, not a number the model was asked to produce. That is a testable claim, so we tested it.

The experiment

We exported the pipeline's own records with their regex labels — every label clause the polarity engine had classified, the top 80 ranked datasets for each disease, and the trials it had kept or dropped — and sent a random sample of each (572, 400 and 400 records) to Jev with the questions above. The regex label is not ground truth; it is the other opinion. Where the two disagreed, a person read the record.

The calibration held. Binning every answer by its reported confidence:

On the cohort task the same table looks worse — 73% agreement even at confidence 1.0 — but there the disagreements are the regex being wrong, which is the point of the exercise: calibration is against the truth, not against the other classifier, and the truth had to be read by hand.

That last row is the useful one. A classifier that says I don't know when it doesn't, and is right when it says it is sure, can be wired into a pipeline with a threshold. A chat model asked for a confidence score gives a number; it is not this number.

The rule we run

The regex stays the base classifier. Without a key, or with the API down, the pipeline produces exactly what it did before. Jev is a second opinion, combined by a fixed rule:

regex == jev                     → keep
differ, jev probability ≥ 0.85  → take jev's label, log the change
differ, below                    → keep regex, flag needs_review

On the first 100 neuroblastoma datasets: 67 agreed, 15 were changed by Jev, 18 were flagged. All 15 changes held up on reading. A few of them:

The last row is the threshold doing its job from the other side. Jev was right — a PDX is a mouse — but at 0.83 it did not get to overrule the regex on its own; a person did, in thirty seconds, from the review list. The cohort tag feeds the dataset score directly (patient material earns the largest bonus), so each of these changes moves a series up or down the page a reader actually sees.

What it cost

1,374 requests, 801,259 input tokens, $0.031, at $0.042 per million input tokens with output free. Each call took about a fifth of a second. The full run over all 26,000 ranked datasets is roughly half a dollar; re-classifying only new records each week is cents. The same 1,374 calls through a general chat model would have been somewhere between one and twenty dollars depending on the model, and hours rather than minutes — affordable once, not weekly across 53 diseases and three tasks. The ratio matters more than the absolute number, and the calibration matters more than the ratio.

Where this goes next

• Cohort tags on every briefing — the full dry run is finishing as this is written; the corrections apply to the dataset rankings on the next rebuild, and the review list is published in each disease's JSON as needs_review.
• Polarity with context — the clause plus the clauses before it, so Jev sees what the regex sees, and several questions per call (polarity, role, whether the indication is restricted to a biomarker) at the same price.
• Cross-tumour reasoning — when a drug is labelled for gene X in cancer A and cancer B carries the same gene, the question is not whether X is there but whether the same alteration, co-mutations, dependency and expression context are. Each of those is a fixed-option question over structured evidence, which is what a decision model is for. The evidence layer — variant-level alteration and co-alteration tables from clinical sequencing cohorts — is being built now.
• The jobs board — the same pattern (regex relevance score, calibrated second opinion, review list) applies to deciding which of a few thousand postings a day are research roles, and to tagging the level the board's readers actually filter on.

Frequently asked questions

Related

• Briefings methods — how each section is built, what is read and rejected, and the limits.
• Neuroblastoma briefing — the page the examples above come from.
• How to find CAR-T targets and trials for any gene — another place where the obvious index misses the evidence.