Suppose a model decides which items to accept automatically and which to send to a reviewer. You set a threshold at 90% confidence. That number describes a useful policy only if the accepted answers are actually right often enough for the work you're doing.
We tested that assumption on Jev, a decision model that returns structured answers and probabilities. Its confidence was useful for ranking answers, but the raw values overstated how often they were correct. The size of that error also changed with the way we asked the question.
What we measured
The Jev-Calibration experiment used 8,801 distinct sentiment examples after removing exact duplicates from a 10,000-line dataset. The examples included strong, medium, weak, and neutral sentiment. We gave the 2,000 neutral examples deliberately arbitrary labels, so we expected chance-level accuracy for that tier. We reported results with and without it.
We used Jev version jev-1.13.0 and split the data into 5,280 calibration examples and 3,521 untouched test examples. One setup asked a yes-or-no question, Jev's Noul type, about positive sentiment. Another used a Choice question with positive and negative options. Further setups changed the wording, polarity, and options.
Across the full dataset, Noul's mean confidence was 79.0%, while accuracy was 72.3%. Choice's mean confidence was 91.4%, while accuracy was 76.1%. These averages already make a raw 90% routing threshold difficult to interpret. The confidence buckets showed a more specific problem: for Choice, stated confidence below 95% carried little information about correctness on this data.
Ranking and calibration answer different questions
A useful ranking puts more correct answers near the top. Calibration asks whether the number attached to an answer matches the observed success rate. A model can rank answers well while giving them probabilities that are too high.
We fitted a mapping from raw confidence to observed correctness on the calibration partition, then checked it on the test partition. Platt scaling fits a smooth curve. Isotonic regression fits a mapping that can stay flat or rise as the raw score increases. Neither method adds information to the underlying prediction.
For the main Noul setup, expected calibration error on the test partition fell from 0.117 with raw scores to 0.052 with Platt scaling and 0.008 with isotonic regression. Across eleven setups, isotonic regression reduced that error to between 0.006 and 0.018. Those are results for this dataset and model version. They don't establish the same improvement for another application.
The question is part of the classifier
Changing the phrasing changed both accuracy and calibration. Across the tested setups, accuracy at the default cutoff ranged from 72.3% to 76.9%, and raw calibration error ranged from 6.4 to 16.0 percentage points. Check a confidence mapping fitted for one setup before you reuse it with another.
Check the confidence values against outcomes from the work the threshold will control. Start with labels from your own reviewers, keep evaluation examples separate from fitting, and measure the errors that would remain among automatically accepted items. A sentiment benchmark can explain the method; it can't choose the error rate your business should accept.
Keep calibration with the deployed version
Cyclotron fits a small learned model from reviewer labels and calibrates its confidence. It retains the classifier context and evaluation evidence with the version that produced the predictions. When the rubric, examples, or questions change, the resulting candidate needs its own evaluation.
Agree on the question setup first. Collect representative labels, including difficult cases. Fit on one partition, check on another, and choose a review policy from the observed errors. Recheck it when the model, question, or incoming work changes. The original experiment includes raw responses and charts for inspecting each step; Cyclotron's explanation shows how feedback and evaluation fit into ongoing operation.
