Which decision model, and what it costs

The task is the one behind the example on our home page, but this is an earlier recording of it: 400 Wikinews stories with the earlier label set. The home page example uses a newer label set, so the numbers on the two pages are not comparable. In both, the job is to decide publish or reject for a curated constructive-news feed, with every decision reviewed against the same frozen reference labels. The labels are simulated editorial judgments, so this is a controlled comparison, not a newsroom. Each arm started from an empty rubric, used the same seed and the same gate settings, and used gpt-4.1-mini as the optimizer, so the only thing that changed was the model making the decisions.

Three models took that seat: Jev 1.13.0, a hosted decision model that answers a bounded question with a label and a confidence; gpt-4.1-mini, a small general LLM used as a classifier; and gpt-4.1, a larger general LLM. Costs are list prices applied to the usage each provider reported on every recorded call.

By the fourth hundred stories all three sat in the same place. Jev scored 80, 83, 81, and 78 percent across the four hundreds, gpt-4.1-mini scored 52 percent in its first hundred and 82 in its last, and gpt-4.1 scored 55, 83, 79, and 77. Over the second half of the run Jev averaged 79.5 percent and gpt-4.1 78. gpt-4.1-mini scored 83 on the recording shown here, and 78 averaged over its five recorded runs, from 75 to 83.

The difference is where they started. Jev was right four times in five from the first hundred, with no rubric and no examples. The two general LLMs started near 55 percent and needed the first two hundred corrections to catch up. The larger LLM did not raise the ceiling; it reached the same plateau as the small one. These runs gave every model the full story. The home page example is a newer recording with a different label set, so its numbers cannot be read against these.

The same 400 stories cost five cents of decision-model calls on Jev, about 45 cents on gpt-4.1-mini, and $3.74 on gpt-4.1. Per thousand decisions that is roughly 4 cents, 40 cents, and $2. Jev also finished in 7 minutes against 25 for gpt-4.1.

The optimizer cost about 32 cents in every arm, because it ran the same eight times on the same LLM. In the Jev run, the optimizer was six times the cost of the decisions. That is the shape you want: the expensive model runs rarely, the cheap one runs on everything.

Accuracy was level; confidence was not. Raw Jev grew surer as the run went on, from a stated 84 percent to 94 percent, while its accuracy stayed near 80. The cyclotron’s calibrated confidence stayed at 81 to 86 percent, close to the measured accuracy. Its calibration error in the last hundred was 0.07, against 0.14 for raw Jev and 0.17 for raw gpt-4.1.

That matters for routing. In the last hundred stories, the least confident 15 percent of Jev’s decisions held 41 percent of its mistakes, and the least confident 30 percent held 61 percent. For gpt-4.1 the same slices held 22 and 47 percent. A reviewer who only looks at the least confident Jev decisions catches far more than one who looks at the least confident gpt-4.1 decisions.

Jev was also the only arm where the cyclotron’s learned model changed labels often enough to measure: it overrode the raw model on 28 decisions and was right on 17 of them, against the raw model’s 11.

One corpus, one run per arm, simulated labels, and an optimizer that never promoted a rubric change in any arm. The numbers describe this task at this size, not the models in general. The full notes, fixtures, and per-window tables are in the engine repository under the experiments folder, and the three recordings are in the site’s own repository.

Put a decision model in the decision seat and measure it on your own items before anything else. In this recorded 400-story comparison, Jev was right four times in five in its first hundred. The question then is how to route the fifth decision to a person, and calibrated confidence helps make that routing work.