Recorded runs

Recordings

Each recording is one recorded run. Settings and starting points differ between them, so read the setup before comparing.

To see what the settings change, read the decisions that come first and the editorial triage use case.

Compare runs

How the run starts

What changes when the first rubric arrives sooner, or is written in advance?

Accuracy by window of stories

  • Blank start, default trigger
  • Blank start, early trigger
  • Written starting rubric
  • Rejecting everything
Accuracy by window of storiesAccuracy by window of stories. Blank start, default trigger: 44.3% in stories 1–70, 84.3% in stories 71–140, 90.0% in stories 141–210, 95.7% in stories 211–280. Blank start, early trigger: 77.1% in stories 1–70, 94.3% in stories 71–140, 92.9% in stories 141–210, 95.7% in stories 211–280. Written starting rubric: 91.4% in stories 1–70, 94.3% in stories 71–140, 94.3% in stories 141–210, 92.9% in stories 211–280. Rejecting everything: 84.3% in stories 1–70, 80.0% in stories 71–140, 78.6% in stories 141–210, 84.3% in stories 211–280.0%25%50%75%100%accuracy1–7071–140141–210211–280stories in each windowBlank start, default trigger: 44.3% accurate in stories 1–70. 70 stories reviewed by then.Blank start, default trigger: 84.3% accurate in stories 71–140. 140 stories reviewed by then.Blank start, default trigger: 90.0% accurate in stories 141–210. 210 stories reviewed by then.Blank start, default trigger: 95.7% accurate in stories 211–280. 280 stories reviewed by then.Blank start, early trigger: 77.1% accurate in stories 1–70. 70 stories reviewed by then.Blank start, early trigger: 94.3% accurate in stories 71–140. 140 stories reviewed by then.Blank start, early trigger: 92.9% accurate in stories 141–210. 210 stories reviewed by then.Blank start, early trigger: 95.7% accurate in stories 211–280. 280 stories reviewed by then.Written starting rubric: 91.4% accurate in stories 1–70. 70 stories reviewed by then.Written starting rubric: 94.3% accurate in stories 71–140. 140 stories reviewed by then.Written starting rubric: 94.3% accurate in stories 141–210. 210 stories reviewed by then.Written starting rubric: 92.9% accurate in stories 211–280. 280 stories reviewed by then.
One run each. A window of 70 stories carries about plus or minus 7 points. The dashed grey line is what rejecting everything would have scored in each window (reject is the most common label), from 78.6% to 84.3%.

Accuracy against stories reviewed

  • Blank start, default trigger
  • Blank start, early trigger
  • Written starting rubric
Accuracy against stories reviewedAccuracy against stories reviewed. Blank start, default trigger: 44.3% after 70 stories reviewed, 84.3% after 140 stories reviewed, 90.0% after 210 stories reviewed, 95.7% after 280 stories reviewed. Blank start, early trigger: 77.1% after 70 stories reviewed, 94.3% after 140 stories reviewed, 92.9% after 210 stories reviewed, 95.7% after 280 stories reviewed. Written starting rubric: 91.4% after 70 stories reviewed, 94.3% after 140 stories reviewed, 94.3% after 210 stories reviewed, 92.9% after 280 stories reviewed.0%25%50%75%100%accuracy070140210280stories reviewed so farBlank start, default trigger: 44.3% accurate in stories 1–70, with 70 stories reviewed by then.Blank start, default trigger: 84.3% accurate in stories 71–140, with 140 stories reviewed by then.Blank start, default trigger: 90.0% accurate in stories 141–210, with 210 stories reviewed by then.Blank start, default trigger: 95.7% accurate in stories 211–280, with 280 stories reviewed by then.Blank start, early trigger: 77.1% accurate in stories 1–70, with 70 stories reviewed by then.Blank start, early trigger: 94.3% accurate in stories 71–140, with 140 stories reviewed by then.Blank start, early trigger: 92.9% accurate in stories 141–210, with 210 stories reviewed by then.Blank start, early trigger: 95.7% accurate in stories 211–280, with 280 stories reviewed by then.Written starting rubric: 91.4% accurate in stories 1–70, with 70 stories reviewed by then.Written starting rubric: 94.3% accurate in stories 71–140, with 140 stories reviewed by then.Written starting rubric: 94.3% accurate in stories 141–210, with 210 stories reviewed by then.Written starting rubric: 92.9% accurate in stories 211–280, with 280 stories reviewed by then.
One run each. A window of 70 stories carries about plus or minus 7 points. Each point is the end of one window: how many stories had their label revealed to the cyclotron by then, and the accuracy in that window. Stories reviewed by the end of the run: Blank start, default trigger, 280 of 280; Blank start, early trigger, 280 of 280; Written starting rubric, 280 of 280.

The numbers

The numbers behind both charts: stories reviewed, accuracy in each window, optimizer requests and cost
RunStories reviewedStories 1–70Stories 71–140Stories 141–210Stories 211–280Optimizer requestsCost
Blank start, default trigger280 of 28044.3%84.3%90.0%95.7%3$0.06
Blank start, early trigger280 of 28077.1%94.3%92.9%95.7%13$0.25
Written starting rubric280 of 28091.4%94.3%94.3%92.9%13$0.22
Rejecting everything84.3%80.0%78.6%84.3%

How much gets reviewed

How many stories need a reviewer once the first rubric exists?

Accuracy by window of stories

  • Every story reviewed
  • All for 50 stories, then half
  • Lighter schedule from story 50
  • Rejecting everything
Accuracy by window of storiesAccuracy by window of stories. Every story reviewed: 77.1% in stories 1–70, 94.3% in stories 71–140, 92.9% in stories 141–210, 95.7% in stories 211–280. All for 50 stories, then half: 78.6% in stories 1–70, 97.1% in stories 71–140, 92.9% in stories 141–210, 94.3% in stories 211–280. Lighter schedule from story 50: 78.6% in stories 1–70, 95.7% in stories 71–140, 97.1% in stories 141–210, 91.4% in stories 211–280. Rejecting everything: 84.3% in stories 1–70, 80.0% in stories 71–140, 78.6% in stories 141–210, 84.3% in stories 211–280.0%25%50%75%100%accuracy1–7071–140141–210211–280stories in each windowEvery story reviewed: 77.1% accurate in stories 1–70. 70 stories reviewed by then.Every story reviewed: 94.3% accurate in stories 71–140. 140 stories reviewed by then.Every story reviewed: 92.9% accurate in stories 141–210. 210 stories reviewed by then.Every story reviewed: 95.7% accurate in stories 211–280. 280 stories reviewed by then.All for 50 stories, then half: 78.6% accurate in stories 1–70. 62 stories reviewed by then.All for 50 stories, then half: 97.1% accurate in stories 71–140. 97 stories reviewed by then.All for 50 stories, then half: 92.9% accurate in stories 141–210. 139 stories reviewed by then.All for 50 stories, then half: 94.3% accurate in stories 211–280. 167 stories reviewed by then.Lighter schedule from story 50: 78.6% accurate in stories 1–70. 43 stories reviewed by then.Lighter schedule from story 50: 95.7% accurate in stories 71–140. 65 stories reviewed by then.Lighter schedule from story 50: 97.1% accurate in stories 141–210. 93 stories reviewed by then.Lighter schedule from story 50: 91.4% accurate in stories 211–280. 113 stories reviewed by then.
One run each. A window of 70 stories carries about plus or minus 7 points. The dashed grey line is what rejecting everything would have scored in each window (reject is the most common label), from 78.6% to 84.3%.

Accuracy against stories reviewed

  • Every story reviewed
  • All for 50 stories, then half
  • Lighter schedule from story 50
Accuracy against stories reviewedAccuracy against stories reviewed. Every story reviewed: 77.1% after 70 stories reviewed, 94.3% after 140 stories reviewed, 92.9% after 210 stories reviewed, 95.7% after 280 stories reviewed. All for 50 stories, then half: 78.6% after 62 stories reviewed, 97.1% after 97 stories reviewed, 92.9% after 139 stories reviewed, 94.3% after 167 stories reviewed. Lighter schedule from story 50: 78.6% after 43 stories reviewed, 95.7% after 65 stories reviewed, 97.1% after 93 stories reviewed, 91.4% after 113 stories reviewed.0%25%50%75%100%accuracy070140210280stories reviewed so farEvery story reviewed: 77.1% accurate in stories 1–70, with 70 stories reviewed by then.Every story reviewed: 94.3% accurate in stories 71–140, with 140 stories reviewed by then.Every story reviewed: 92.9% accurate in stories 141–210, with 210 stories reviewed by then.Every story reviewed: 95.7% accurate in stories 211–280, with 280 stories reviewed by then.All for 50 stories, then half: 78.6% accurate in stories 1–70, with 62 stories reviewed by then.All for 50 stories, then half: 97.1% accurate in stories 71–140, with 97 stories reviewed by then.All for 50 stories, then half: 92.9% accurate in stories 141–210, with 139 stories reviewed by then.All for 50 stories, then half: 94.3% accurate in stories 211–280, with 167 stories reviewed by then.Lighter schedule from story 50: 78.6% accurate in stories 1–70, with 43 stories reviewed by then.Lighter schedule from story 50: 95.7% accurate in stories 71–140, with 65 stories reviewed by then.Lighter schedule from story 50: 97.1% accurate in stories 141–210, with 93 stories reviewed by then.Lighter schedule from story 50: 91.4% accurate in stories 211–280, with 113 stories reviewed by then.
One run each. A window of 70 stories carries about plus or minus 7 points. Each point is the end of one window: how many stories had their label revealed to the cyclotron by then, and the accuracy in that window. Stories reviewed by the end of the run: Every story reviewed, 280 of 280; All for 50 stories, then half, 167 of 280; Lighter schedule from story 50, 113 of 280.

The numbers

The numbers behind both charts: stories reviewed, accuracy in each window, optimizer requests and cost
RunStories reviewedStories 1–70Stories 71–140Stories 141–210Stories 211–280Optimizer requestsCost
Every story reviewed280 of 28077.1%94.3%92.9%95.7%13$0.25
All for 50 stories, then half167 of 28078.6%97.1%92.9%94.3%6$0.08
Lighter schedule from story 50113 of 28078.6%95.7%97.1%91.4%5$0.05
Rejecting everything84.3%80.0%78.6%84.3%

Which decision model decides

What changes when a small local model decides instead of the hosted one?

Accuracy by window of stories

  • Jev (hosted)
  • Kev (local)
  • Rejecting everything
Accuracy by window of storiesAccuracy by window of stories. Jev (hosted): 86.5% in stories 1–148, 96.6% in stories 149–296, 94.6% in stories 297–444, 94.6% in stories 445–592, 94.6% in stories 593–740, 98.6% in stories 741–888, 94.6% in stories 889–1036, 98.0% in stories 1037–1184. Kev (local): 87.2% in stories 1–148, 85.1% in stories 149–296, 89.9% in stories 297–444, 86.5% in stories 445–592, 85.8% in stories 593–740, 84.5% in stories 741–888, 87.2% in stories 889–1036, 90.5% in stories 1037–1184. Rejecting everything: 88.5% in stories 1–148, 81.8% in stories 149–296, 88.5% in stories 297–444, 85.8% in stories 445–592, 83.8% in stories 593–740, 82.4% in stories 741–888, 84.5% in stories 889–1036, 85.1% in stories 1037–1184.0%25%50%75%100%accuracy1–148149–296297–444445–592593–740741–888889–10361037–1184stories in each windowJev (hosted): 86.5% accurate in stories 1–148. 148 stories reviewed by then.Jev (hosted): 96.6% accurate in stories 149–296. 296 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 297–444. 444 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 445–592. 592 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 593–740. 740 stories reviewed by then.Jev (hosted): 98.6% accurate in stories 741–888. 888 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 889–1036. 1036 stories reviewed by then.Jev (hosted): 98.0% accurate in stories 1037–1184. 1184 stories reviewed by then.Kev (local): 87.2% accurate in stories 1–148. 148 stories reviewed by then.Kev (local): 85.1% accurate in stories 149–296. 296 stories reviewed by then.Kev (local): 89.9% accurate in stories 297–444. 444 stories reviewed by then.Kev (local): 86.5% accurate in stories 445–592. 592 stories reviewed by then.Kev (local): 85.8% accurate in stories 593–740. 740 stories reviewed by then.Kev (local): 84.5% accurate in stories 741–888. 888 stories reviewed by then.Kev (local): 87.2% accurate in stories 889–1036. 1036 stories reviewed by then.Kev (local): 90.5% accurate in stories 1037–1184. 1184 stories reviewed by then.
One run each. A window of 148 stories carries about plus or minus 5 points. The dashed grey line is what rejecting everything would have scored in each window (reject is the most common label), from 81.8% to 88.5%.

Accuracy against stories reviewed

  • Jev (hosted)
  • Kev (local)
Accuracy against stories reviewedAccuracy against stories reviewed. Jev (hosted): 86.5% after 148 stories reviewed, 96.6% after 296 stories reviewed, 94.6% after 444 stories reviewed, 94.6% after 592 stories reviewed, 94.6% after 740 stories reviewed, 98.6% after 888 stories reviewed, 94.6% after 1036 stories reviewed, 98.0% after 1184 stories reviewed. Kev (local): 87.2% after 148 stories reviewed, 85.1% after 296 stories reviewed, 89.9% after 444 stories reviewed, 86.5% after 592 stories reviewed, 85.8% after 740 stories reviewed, 84.5% after 888 stories reviewed, 87.2% after 1036 stories reviewed, 90.5% after 1184 stories reviewed.0%25%50%75%100%accuracy014829644459274088810361184stories reviewed so farJev (hosted): 86.5% accurate in stories 1–148, with 148 stories reviewed by then.Jev (hosted): 96.6% accurate in stories 149–296, with 296 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 297–444, with 444 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 445–592, with 592 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 593–740, with 740 stories reviewed by then.Jev (hosted): 98.6% accurate in stories 741–888, with 888 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 889–1036, with 1036 stories reviewed by then.Jev (hosted): 98.0% accurate in stories 1037–1184, with 1184 stories reviewed by then.Kev (local): 87.2% accurate in stories 1–148, with 148 stories reviewed by then.Kev (local): 85.1% accurate in stories 149–296, with 296 stories reviewed by then.Kev (local): 89.9% accurate in stories 297–444, with 444 stories reviewed by then.Kev (local): 86.5% accurate in stories 445–592, with 592 stories reviewed by then.Kev (local): 85.8% accurate in stories 593–740, with 740 stories reviewed by then.Kev (local): 84.5% accurate in stories 741–888, with 888 stories reviewed by then.Kev (local): 87.2% accurate in stories 889–1036, with 1036 stories reviewed by then.Kev (local): 90.5% accurate in stories 1037–1184, with 1184 stories reviewed by then.
One run each. A window of 148 stories carries about plus or minus 5 points. Each point is the end of one window: how many stories had their label revealed to the cyclotron by then, and the accuracy in that window. Stories reviewed by the end of the run: Jev (hosted), 1184 of 1184; Kev (local), 1184 of 1184.

The numbers

The numbers behind both charts: stories reviewed, accuracy in each window, optimizer requests and cost
RunStories reviewedStories 1–148Stories 149–296Stories 297–444Stories 445–592Stories 593–740Stories 741–888Stories 889–1036Stories 1037–1184Optimizer requestsCost
Jev (hosted)1184 of 118486.5%96.6%94.6%94.6%94.6%98.6%94.6%98.0%20$0.66
Kev (local)1184 of 118487.2%85.1%89.9%86.5%85.8%84.5%87.2%90.5%20$0.31
Rejecting everything88.5%81.8%88.5%85.8%83.8%82.4%84.5%85.1%

Featured · on the home page

Blank start: the first rubric lifts accuracy

A cyclotron starts with no written rubric and learns one from reviewer labels as 280 stories arrive. The first rubric proposal waits for 20 label transitions, so it arrives after about 80 stories.

Setup of Blank start: the first rubric lifts accuracy
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 20 cycles
First rubricCycle 82
0%25%50%75%100%70140210280stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.06 in model calls at list prices: 738 decision-model calls ($0.04) and 3 optimizer requests ($0.02).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric; an offline audit found it over-published against that rubric. These v3 labels come from two independent blind relabeling passes under an explicit written policy; only stories where both passes agreed are included.

Kev as the decision model on 1,186 stories

The same study with Kev, a small model that runs locally at no per-call cost, deciding instead of Jev. The Kev revision is pinned.

Setup of Kev as the decision model on 1,186 stories
Starting rubricBlank (no written rubric)
Decision modelKev
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 6 cycles
First rubricCycle 35
0%25%50%75%100%14829644459274088810361184stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.31 in model calls at list prices: 6,607 decision-model calls ($0.00) and 20 optimizer requests ($0.31).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric. These labels come from two independent blind relabeling passes under an explicit written policy; where the passes disagreed (about 3 percent of stories), the label was settled by hand.

Blank start on 853 stories

Cyclotron starts with no rubric and learns one while deciding 853 stories in order, with every label revealed after each decision.

Setup of Blank start on 853 stories
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 6 cycles
First rubricCycle 24
0%25%50%75%100%142284426568710852stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.64 in model calls at list prices: 5,477 decision-model calls ($0.38) and 20 optimizer requests ($0.26).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric. These labels come from two independent blind relabeling passes under an explicit written policy; where the passes disagreed (6 percent of the new stories), the label was settled by hand.

Blank start, first rubric after 21 stories

Same setup as the first recording, but the first rubric is proposed after 6 label transitions instead of 20. The loop writes its first rubric at story 21 and gets about 13 optimizer calls instead of 4.

Setup of Blank start, first rubric after 21 stories
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 6 cycles
First rubricCycle 21
0%25%50%75%100%70140210280stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.25 in model calls at list prices: 2,313 decision-model calls ($0.15) and 13 optimizer requests ($0.10).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric; an offline audit found it over-published against that rubric. These v3 labels come from two independent blind relabeling passes under an explicit written policy; only stories where both passes agreed are included.

Review everything first, then half

Same blank start and early trigger as the full-review run, but after the first 50 stories only half of the stories are reviewed.

Setup of Review everything first, then half
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewed167 of 280 stories (every story for the first 50 stories, then half of the stories)
Rubric changes consideredEvery 6 cycles
First rubricCycle 21
0%25%50%75%100%70140210280stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.08 in model calls at list prices: 904 decision-model calls ($0.04) and 6 optimizer requests ($0.03).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric; an offline audit found it over-published against that rubric. These v3 labels come from two independent blind relabeling passes under an explicit written policy; only stories where both passes agreed are included.

Review less, starting with the rejects

Same blank start and early trigger, but the review schedule is lighter: for the first 50 stories every story the model calls publish and half of those it calls reject are reviewed, then half of the publishes and a quarter of the rejects.

Setup of Review less, starting with the rejects
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewed113 of 280 stories (for the first 50 stories, every story the model calls publish and half of those it calls reject; after that, half of the publishes and a quarter of the rejects)
Rubric changes consideredEvery 6 cycles
First rubricCycle 21
0%25%50%75%100%70140210280stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.05 in model calls at list prices: 637 decision-model calls ($0.03) and 5 optimizer requests ($0.02).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric; an offline audit found it over-published against that rubric. These v3 labels come from two independent blind relabeling passes under an explicit written policy; only stories where both passes agreed are included.

Starting from a written rubric

Instead of a blank start, the cyclotron begins with a written rubric describing what counts as publishable, and the first rubric proposal comes after 6 label transitions.

Setup of Starting from a written rubric
Starting rubricCurated by hand
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 6 cycles
First rubric changeCycle 121
0%25%50%75%100%70140210280stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.22 in model calls at list prices: 2,317 decision-model calls ($0.12) and 13 optimizer requests ($0.10).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric; an offline audit found it over-published against that rubric. These v3 labels come from two independent blind relabeling passes under an explicit written policy; only stories where both passes agreed are included.

Jev as the decision model on 1,186 stories

Cyclotron with Jev deciding, starting from no rubric, learning one over 1,186 stories with every label revealed after each decision.

Setup of Jev as the decision model on 1,186 stories
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 6 cycles
First rubricCycle 35
0%25%50%75%100%14829644459274088810361184stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.66 in model calls at list prices: 5,365 decision-model calls ($0.36) and 20 optimizer requests ($0.30).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric. These labels come from two independent blind relabeling passes under an explicit written policy; where the passes disagreed (about 3 percent of stories), the label was settled by hand.

Recorded run · curated Wikinews editorial feed

280 editorial decisions, replayed in order.

Curated marketing demo using simulated semantic editorial labels; not a representative benchmark or real newsroom votes. Labels are simulated: two independent blind passes under a written policy; stories where the passes disagreed were left out. Cyclotron predicted publish or reject before each label was revealed. In the final 140 recorded requests, the raw decision model and Cyclotron's added final prediction were scored on the same requests: 91.4% and 92.9% accuracy, where rejecting everything would score 81.4%. Their confidence error was Brier 0.140006 and 0.105541. This sequential replay has no independent holdout, and the raw model's rubric context evolved during the run. The whole run cost about $0.06 in model calls at list prices and took about 3 minutes: 738 decision-model calls and 3 optimizer calls. The decision model is Jev; its 738 calls cost about $0.04 at list price. The LLM optimizer is gpt-6-luna; its 3 requests cost about $0.02.

Editorial rule: Publish articles with a meaningful constructive or public-benefit outcome; reject harm-dominant or routine items.

Stories replayed
280
Feedback policy
Review every decision
Windows
Stories are scored in order, in windows of equal size
4 of 70
Accuracy, first → last window
Rejecting everything would score 84% and 84%
44% → 96%
Matched window
Same sequential prediction events
Final 140 stories
Raw → final accuracy
Of all stories, how many it got right. Rejecting everything would score 81.4%.
91.4% → 92.9%

Model costs: list prices, USD per million tokens, applied to the provider-reported usage on every recorded call. Optimizer price (gpt-6-luna): LiteLLM model price table (standard tier), read 2026-10-11.

The recorded animation on the home page uses stories from this same run. Step through all 280 decisions in order, with the recorded labels, confidence, and running metrics.

The labels are simulated editorial labels on a curated public corpus. This shows the recorded replay, not a live newsroom. Full run and limits.

Source: malteos/wikinews at b4c2ec3857fcac203c40b8d61586e934ed07c128; CC BY 2.5 (Wikinews policy; articles dated through 2024-12-15). License details.

The numbers

What moved, and what it cost.

Final 140 recorded decisionsDecision modelAlone, raw decisionCyclotronDecision plus ML modelCorrect decisionsAccuracy · rejecting everything would score 81.4%91.4%92.9%Confidence errorBrier score0.1400060.105541Confidence gapExpected calibration error0.0861430.028031Useful publishesPrecision73.3%94.4%Useful stories foundRecall84.6%65.4%With a reviewer on the least-sure 20%The whole system is correct this often: 92.9% → 97.9%97.9%7 raw mistakes became correct; 5 new mistakes were introduced.Simulated-feedback replay · sequential · no independent holdout

The recorded run starts from a blank rubric. The cyclotron was right 44% of the time in the first 70 stories and 96% in the last 70; rejecting everything would score 84% and 84%. On the same final 140 recorded requests, Cyclotron's final prediction raised accuracy from 91.4% to 92.9%, where rejecting everything would score 81.4%. In the middle windows the decision model's own answers under the learned rubric were as accurate or more accurate than the final prediction (88.6% against 84.3%, then 94.3% against 90.0%). The ML model's gain shows in the last window (88.6% to 95.7%). It found 65.4% of the worthwhile stories (raw 84.6%), at 94.4% publish precision (raw 73.3%). The raw model's rubric context evolved through this sequential simulated-feedback replay. It has no independent holdout and is not a comparison of two identical frozen model versions. Labels are simulated: two independent blind passes under a written policy; stories where the passes disagreed were left out.

Recorded replay, with limits

From blank to right, with confidence that matches results.

rule: approve anything at least 80% sure; people check the rest.approved, rightapproved, wrongchecked by peopledecision model alonechecked0promised 91% right · got 98%whole system right:69 of 70people check 27 of 70; itpromised 91% right and got 98%.with the cyclotronchecked0promised 98% right · got 97%whole system right:68 of 70people check 8 of 70; itpromised 98% and got 97%.

This simulated-feedback replay starts from a blank rubric on 280 Wikinews stories. The LLM optimizer wrote its first rubric after about 80 labeled stories. Its confidence error (ECE) fell from 0.269 to 0.036 across the 4 windows of 70.

On the final matched requests, confidence error fell from Brier 0.140006 to 0.105541 and ECE 0.086143 to 0.028031. In the middle windows the decision model's own answers under the learned rubric were as accurate or more accurate than the final prediction (88.6% against 84.3%, then 94.3% against 90.0%). The ML model's gain shows in the last window (88.6% to 95.7%). With a reviewer taking the least-sure 20% of requests, the whole system was correct 97.9% of the time, up from 92.9%.

Each request was scored before its own simulated feedback was revealed. The LLM optimizer made 3 rubric proposals (2 in window 2, 1 in window 3). The cyclotron put each to use provisionally. An ML model was fitted at about cycle 200 and accepted. The replay is sequential and has no independent holdout; the raw model's rubric context also evolved, so this is not a frozen-base-model before-and-after or a production-savings claim. Labels are simulated: two independent blind passes under a written policy; stories where the passes disagreed were left out.

The routing illustration counts every item checked by an editor as correctly resolved by that editor. Its 68-in-70 whole-system figure is an assumption for the illustration, not a production saving claim. For scale, rejecting everything would be right on 59 of those 70.

“When your business uses a decision model, you remain responsible for the decisions it informs and for mitigating the risks they create.”
Ryan Porter, Who Is Responsible After You Deploy a Decision Model?

Research and evidence · Accountability

Who Is Responsible After You Deploy a Decision Model?

When your business uses a decision model, you remain responsible for the decisions it informs and for mitigating the risks they create.

Read the articleAll research articles