We described a fair six-sided die that had been rolled out of sight and asked Jev which face showed. It chose one and returned P(1) of 88%. The prompt supplied no observation about the roll. Every face still had the same one-in-six chance.
That mismatch is useful for anyone deploying a decision model. A returned probability can be a strong model preference without being a correct estimate of an event's likelihood. The way to tell is to test it where the answer or distribution is known.
Move the answer around the list
One was the first option in the initial request, so we tested whether the model was simply choosing the first entry. Six options have 720 possible orders. We sent every order with digit labels, then word labels, and repeated both sets. Each model received 2,880 requests.
Jev chose one in all 2,880. It followed the answer through every list position. In the first digit pass, its mean P(1) was 89.7%, and mean P(6) was 4.4%. A first-position shortcut couldn't explain that pattern.
The public experiment preserves the preregistration, exact requests, replies, and summaries. Jev's hosted responses identified version jev-1.13.0. Kev and Laya used pinned local builds. The test compared returned choice probabilities, without assuming that separate confidence fields meant the probability of correctness.
Different models had different preferences
Across the same matrix, Kev chose one in 1,410 of 2,880 replies, or 49.0%. Laya chose six in 2,610, or 90.6%. Kev's mean probabilities in the first digit pass were comparatively even: P(1) was 18.6% and P(6) was 18.4%. Laya's means were 6.5% and 66.2% respectively.
Choice order and wording mattered too. Kev chose the first-listed option in 426 of 720 orders in the first digit pass. The released summary records zero last-position selections for Laya in that pass. Replacing digits with words changed Kev's selected answer in 324 paired orders and Laya's in 129. Jev continued to choose one.
Exact repeat requests kept the selected face unchanged for all three models. Repeating an answer consistently tells you about stability; it doesn't establish correctness. Those repeated requests were also not additional die rolls. They measured how the models responded to the same information.
The cause is still open
We began with a hypothesis about Benford's Law, the unequal frequency of leading digits in some numerical datasets. Perhaps a learned frequency pattern encouraged Jev to favor one. We didn't inspect its training corpus, count its numeral tokens, or change those frequencies through retraining. The experiment measures the output preference and leaves its cause unresolved.
Kev's partial preference for one and Laya's preference for six also resist a single explanation. Treating a pattern in the answers as a confirmed account of training would claim more than the evidence supports.
Test the request before using its probabilities
For a production classifier, preserve the exact question, option descriptions, ordering, and model version with the measurement. Check equivalent wording and reordered options. Then compare the scores with labelled outcomes from the work you intend to automate. A raw probability shouldn't become a routing threshold simply because it looks precise.
Cyclotron retains classifier context, learned versions, and evaluation results together. That lets a team compare a changed question with the version it would replace. The calibration study explains how a useful confidence ranking can still overstate correctness, and how an outcome-based mapping can improve it.
Use a random-number generator when the requirement is a fair random choice. For decisions based on text, start with the released requests and results, reproduce the checks that matter for your application, and measure the complete classifier before using it to control a workflow.
