All use cases
Recorded

Editorial triage

Should this story be published or rejected?

How we know: Three published recordings. Replay each one below.

Who makes this decision today

An editor reads each candidate story and decides. Most stories in the stream are rejects, so the work is finding the few that should run.

What a correction looks like

The cyclotron shows its decision and how sure it is. An editor who disagrees records the other label, which is a correction, and can write the reason. In the recordings the reviewer is simulated: the labels come from a relabeled set of Wikinews stories.

What the settings changed

We recorded three runs on the same 287 stories, with the same decision model and optimizer. They differ in two settings: how the loop starts, and when it may first change its rubric. Each is one run, not a repeated measurement.

With a blank start and the default trigger, accuracy in the first window of 70 stories was 44 percent. With a blank start and an early trigger it was 77 percent. With a written starting rubric it was 91 percent. Rejecting everything would have scored 84 percent in that window.

The first rubric arrived at story 82 with the default trigger and at story 21 with the early trigger. The written start did not need an early rubric: its first rubric change came at story 121.

The final windows were about 96, 96 and 93 percent. A window of 70 stories is noisy, so we do not rank the three on the last window. The runs cost about 6, 25 and 22 cents in model calls at list prices. The early trigger made 13 optimizer requests against 3.

How the run starts

What changes when the first rubric arrives sooner, or is written in advance?

Accuracy by window of stories

  • Blank start, default trigger
  • Blank start, early trigger
  • Written starting rubric
  • Rejecting everything
Accuracy by window of storiesAccuracy by window of stories. Blank start, default trigger: 44.3% in stories 1–70, 84.3% in stories 71–140, 90.0% in stories 141–210, 95.7% in stories 211–280. Blank start, early trigger: 77.1% in stories 1–70, 94.3% in stories 71–140, 92.9% in stories 141–210, 95.7% in stories 211–280. Written starting rubric: 91.4% in stories 1–70, 94.3% in stories 71–140, 94.3% in stories 141–210, 92.9% in stories 211–280. Rejecting everything: 84.3% in stories 1–70, 80.0% in stories 71–140, 78.6% in stories 141–210, 84.3% in stories 211–280.0%25%50%75%100%accuracy1–7071–140141–210211–280stories in each windowBlank start, default trigger: 44.3% accurate in stories 1–70. 70 stories reviewed by then.Blank start, default trigger: 84.3% accurate in stories 71–140. 140 stories reviewed by then.Blank start, default trigger: 90.0% accurate in stories 141–210. 210 stories reviewed by then.Blank start, default trigger: 95.7% accurate in stories 211–280. 280 stories reviewed by then.Blank start, early trigger: 77.1% accurate in stories 1–70. 70 stories reviewed by then.Blank start, early trigger: 94.3% accurate in stories 71–140. 140 stories reviewed by then.Blank start, early trigger: 92.9% accurate in stories 141–210. 210 stories reviewed by then.Blank start, early trigger: 95.7% accurate in stories 211–280. 280 stories reviewed by then.Written starting rubric: 91.4% accurate in stories 1–70. 70 stories reviewed by then.Written starting rubric: 94.3% accurate in stories 71–140. 140 stories reviewed by then.Written starting rubric: 94.3% accurate in stories 141–210. 210 stories reviewed by then.Written starting rubric: 92.9% accurate in stories 211–280. 280 stories reviewed by then.
One run each. A window of 70 stories carries about plus or minus 7 points. The dashed grey line is what rejecting everything would have scored in each window (reject is the most common label), from 78.6% to 84.3%.

Accuracy against stories reviewed

  • Blank start, default trigger
  • Blank start, early trigger
  • Written starting rubric
Accuracy against stories reviewedAccuracy against stories reviewed. Blank start, default trigger: 44.3% after 70 stories reviewed, 84.3% after 140 stories reviewed, 90.0% after 210 stories reviewed, 95.7% after 280 stories reviewed. Blank start, early trigger: 77.1% after 70 stories reviewed, 94.3% after 140 stories reviewed, 92.9% after 210 stories reviewed, 95.7% after 280 stories reviewed. Written starting rubric: 91.4% after 70 stories reviewed, 94.3% after 140 stories reviewed, 94.3% after 210 stories reviewed, 92.9% after 280 stories reviewed.0%25%50%75%100%accuracy070140210280stories reviewed so farBlank start, default trigger: 44.3% accurate in stories 1–70, with 70 stories reviewed by then.Blank start, default trigger: 84.3% accurate in stories 71–140, with 140 stories reviewed by then.Blank start, default trigger: 90.0% accurate in stories 141–210, with 210 stories reviewed by then.Blank start, default trigger: 95.7% accurate in stories 211–280, with 280 stories reviewed by then.Blank start, early trigger: 77.1% accurate in stories 1–70, with 70 stories reviewed by then.Blank start, early trigger: 94.3% accurate in stories 71–140, with 140 stories reviewed by then.Blank start, early trigger: 92.9% accurate in stories 141–210, with 210 stories reviewed by then.Blank start, early trigger: 95.7% accurate in stories 211–280, with 280 stories reviewed by then.Written starting rubric: 91.4% accurate in stories 1–70, with 70 stories reviewed by then.Written starting rubric: 94.3% accurate in stories 71–140, with 140 stories reviewed by then.Written starting rubric: 94.3% accurate in stories 141–210, with 210 stories reviewed by then.Written starting rubric: 92.9% accurate in stories 211–280, with 280 stories reviewed by then.
One run each. A window of 70 stories carries about plus or minus 7 points. Each point is the end of one window: how many stories had their label revealed to the cyclotron by then, and the accuracy in that window. Stories reviewed by the end of the run: Blank start, default trigger, 280 of 280; Blank start, early trigger, 280 of 280; Written starting rubric, 280 of 280.

The numbers

The numbers behind both charts: stories reviewed, accuracy in each window, optimizer requests and cost
RunStories reviewedStories 1–70Stories 71–140Stories 141–210Stories 211–280Optimizer requestsCost
Blank start, default trigger280 of 28044.3%84.3%90.0%95.7%3$0.06
Blank start, early trigger280 of 28077.1%94.3%92.9%95.7%13$0.25
Written starting rubric280 of 28091.4%94.3%94.3%92.9%13$0.22
Rejecting everything84.3%80.0%78.6%84.3%

How much gets reviewed

How many stories need a reviewer once the first rubric exists?

Accuracy by window of stories

  • Every story reviewed
  • All for 50 stories, then half
  • Lighter schedule from story 50
  • Rejecting everything
Accuracy by window of storiesAccuracy by window of stories. Every story reviewed: 77.1% in stories 1–70, 94.3% in stories 71–140, 92.9% in stories 141–210, 95.7% in stories 211–280. All for 50 stories, then half: 78.6% in stories 1–70, 97.1% in stories 71–140, 92.9% in stories 141–210, 94.3% in stories 211–280. Lighter schedule from story 50: 78.6% in stories 1–70, 95.7% in stories 71–140, 97.1% in stories 141–210, 91.4% in stories 211–280. Rejecting everything: 84.3% in stories 1–70, 80.0% in stories 71–140, 78.6% in stories 141–210, 84.3% in stories 211–280.0%25%50%75%100%accuracy1–7071–140141–210211–280stories in each windowEvery story reviewed: 77.1% accurate in stories 1–70. 70 stories reviewed by then.Every story reviewed: 94.3% accurate in stories 71–140. 140 stories reviewed by then.Every story reviewed: 92.9% accurate in stories 141–210. 210 stories reviewed by then.Every story reviewed: 95.7% accurate in stories 211–280. 280 stories reviewed by then.All for 50 stories, then half: 78.6% accurate in stories 1–70. 62 stories reviewed by then.All for 50 stories, then half: 97.1% accurate in stories 71–140. 97 stories reviewed by then.All for 50 stories, then half: 92.9% accurate in stories 141–210. 139 stories reviewed by then.All for 50 stories, then half: 94.3% accurate in stories 211–280. 167 stories reviewed by then.Lighter schedule from story 50: 78.6% accurate in stories 1–70. 43 stories reviewed by then.Lighter schedule from story 50: 95.7% accurate in stories 71–140. 65 stories reviewed by then.Lighter schedule from story 50: 97.1% accurate in stories 141–210. 93 stories reviewed by then.Lighter schedule from story 50: 91.4% accurate in stories 211–280. 113 stories reviewed by then.
One run each. A window of 70 stories carries about plus or minus 7 points. The dashed grey line is what rejecting everything would have scored in each window (reject is the most common label), from 78.6% to 84.3%.

Accuracy against stories reviewed

  • Every story reviewed
  • All for 50 stories, then half
  • Lighter schedule from story 50
Accuracy against stories reviewedAccuracy against stories reviewed. Every story reviewed: 77.1% after 70 stories reviewed, 94.3% after 140 stories reviewed, 92.9% after 210 stories reviewed, 95.7% after 280 stories reviewed. All for 50 stories, then half: 78.6% after 62 stories reviewed, 97.1% after 97 stories reviewed, 92.9% after 139 stories reviewed, 94.3% after 167 stories reviewed. Lighter schedule from story 50: 78.6% after 43 stories reviewed, 95.7% after 65 stories reviewed, 97.1% after 93 stories reviewed, 91.4% after 113 stories reviewed.0%25%50%75%100%accuracy070140210280stories reviewed so farEvery story reviewed: 77.1% accurate in stories 1–70, with 70 stories reviewed by then.Every story reviewed: 94.3% accurate in stories 71–140, with 140 stories reviewed by then.Every story reviewed: 92.9% accurate in stories 141–210, with 210 stories reviewed by then.Every story reviewed: 95.7% accurate in stories 211–280, with 280 stories reviewed by then.All for 50 stories, then half: 78.6% accurate in stories 1–70, with 62 stories reviewed by then.All for 50 stories, then half: 97.1% accurate in stories 71–140, with 97 stories reviewed by then.All for 50 stories, then half: 92.9% accurate in stories 141–210, with 139 stories reviewed by then.All for 50 stories, then half: 94.3% accurate in stories 211–280, with 167 stories reviewed by then.Lighter schedule from story 50: 78.6% accurate in stories 1–70, with 43 stories reviewed by then.Lighter schedule from story 50: 95.7% accurate in stories 71–140, with 65 stories reviewed by then.Lighter schedule from story 50: 97.1% accurate in stories 141–210, with 93 stories reviewed by then.Lighter schedule from story 50: 91.4% accurate in stories 211–280, with 113 stories reviewed by then.
One run each. A window of 70 stories carries about plus or minus 7 points. Each point is the end of one window: how many stories had their label revealed to the cyclotron by then, and the accuracy in that window. Stories reviewed by the end of the run: Every story reviewed, 280 of 280; All for 50 stories, then half, 167 of 280; Lighter schedule from story 50, 113 of 280.

The numbers

The numbers behind both charts: stories reviewed, accuracy in each window, optimizer requests and cost
RunStories reviewedStories 1–70Stories 71–140Stories 141–210Stories 211–280Optimizer requestsCost
Every story reviewed280 of 28077.1%94.3%92.9%95.7%13$0.25
All for 50 stories, then half167 of 28078.6%97.1%92.9%94.3%6$0.08
Lighter schedule from story 50113 of 28078.6%95.7%97.1%91.4%5$0.05
Rejecting everything84.3%80.0%78.6%84.3%

Which decision model decides

What changes when a small local model decides instead of the hosted one?

Accuracy by window of stories

  • Jev (hosted)
  • Kev (local)
  • Rejecting everything
Accuracy by window of storiesAccuracy by window of stories. Jev (hosted): 86.5% in stories 1–148, 96.6% in stories 149–296, 94.6% in stories 297–444, 94.6% in stories 445–592, 94.6% in stories 593–740, 98.6% in stories 741–888, 94.6% in stories 889–1036, 98.0% in stories 1037–1184. Kev (local): 87.2% in stories 1–148, 85.1% in stories 149–296, 89.9% in stories 297–444, 86.5% in stories 445–592, 85.8% in stories 593–740, 84.5% in stories 741–888, 87.2% in stories 889–1036, 90.5% in stories 1037–1184. Rejecting everything: 88.5% in stories 1–148, 81.8% in stories 149–296, 88.5% in stories 297–444, 85.8% in stories 445–592, 83.8% in stories 593–740, 82.4% in stories 741–888, 84.5% in stories 889–1036, 85.1% in stories 1037–1184.0%25%50%75%100%accuracy1–148149–296297–444445–592593–740741–888889–10361037–1184stories in each windowJev (hosted): 86.5% accurate in stories 1–148. 148 stories reviewed by then.Jev (hosted): 96.6% accurate in stories 149–296. 296 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 297–444. 444 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 445–592. 592 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 593–740. 740 stories reviewed by then.Jev (hosted): 98.6% accurate in stories 741–888. 888 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 889–1036. 1036 stories reviewed by then.Jev (hosted): 98.0% accurate in stories 1037–1184. 1184 stories reviewed by then.Kev (local): 87.2% accurate in stories 1–148. 148 stories reviewed by then.Kev (local): 85.1% accurate in stories 149–296. 296 stories reviewed by then.Kev (local): 89.9% accurate in stories 297–444. 444 stories reviewed by then.Kev (local): 86.5% accurate in stories 445–592. 592 stories reviewed by then.Kev (local): 85.8% accurate in stories 593–740. 740 stories reviewed by then.Kev (local): 84.5% accurate in stories 741–888. 888 stories reviewed by then.Kev (local): 87.2% accurate in stories 889–1036. 1036 stories reviewed by then.Kev (local): 90.5% accurate in stories 1037–1184. 1184 stories reviewed by then.
One run each. A window of 148 stories carries about plus or minus 5 points. The dashed grey line is what rejecting everything would have scored in each window (reject is the most common label), from 81.8% to 88.5%.

Accuracy against stories reviewed

  • Jev (hosted)
  • Kev (local)
Accuracy against stories reviewedAccuracy against stories reviewed. Jev (hosted): 86.5% after 148 stories reviewed, 96.6% after 296 stories reviewed, 94.6% after 444 stories reviewed, 94.6% after 592 stories reviewed, 94.6% after 740 stories reviewed, 98.6% after 888 stories reviewed, 94.6% after 1036 stories reviewed, 98.0% after 1184 stories reviewed. Kev (local): 87.2% after 148 stories reviewed, 85.1% after 296 stories reviewed, 89.9% after 444 stories reviewed, 86.5% after 592 stories reviewed, 85.8% after 740 stories reviewed, 84.5% after 888 stories reviewed, 87.2% after 1036 stories reviewed, 90.5% after 1184 stories reviewed.0%25%50%75%100%accuracy014829644459274088810361184stories reviewed so farJev (hosted): 86.5% accurate in stories 1–148, with 148 stories reviewed by then.Jev (hosted): 96.6% accurate in stories 149–296, with 296 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 297–444, with 444 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 445–592, with 592 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 593–740, with 740 stories reviewed by then.Jev (hosted): 98.6% accurate in stories 741–888, with 888 stories reviewed by then.Jev (hosted): 94.6% accurate in stories 889–1036, with 1036 stories reviewed by then.Jev (hosted): 98.0% accurate in stories 1037–1184, with 1184 stories reviewed by then.Kev (local): 87.2% accurate in stories 1–148, with 148 stories reviewed by then.Kev (local): 85.1% accurate in stories 149–296, with 296 stories reviewed by then.Kev (local): 89.9% accurate in stories 297–444, with 444 stories reviewed by then.Kev (local): 86.5% accurate in stories 445–592, with 592 stories reviewed by then.Kev (local): 85.8% accurate in stories 593–740, with 740 stories reviewed by then.Kev (local): 84.5% accurate in stories 741–888, with 888 stories reviewed by then.Kev (local): 87.2% accurate in stories 889–1036, with 1036 stories reviewed by then.Kev (local): 90.5% accurate in stories 1037–1184, with 1184 stories reviewed by then.
One run each. A window of 148 stories carries about plus or minus 5 points. Each point is the end of one window: how many stories had their label revealed to the cyclotron by then, and the accuracy in that window. Stories reviewed by the end of the run: Jev (hosted), 1184 of 1184; Kev (local), 1184 of 1184.

The numbers

The numbers behind both charts: stories reviewed, accuracy in each window, optimizer requests and cost
RunStories reviewedStories 1–148Stories 149–296Stories 297–444Stories 445–592Stories 593–740Stories 741–888Stories 889–1036Stories 1037–1184Optimizer requestsCost
Jev (hosted)1184 of 118486.5%96.6%94.6%94.6%94.6%98.6%94.6%98.0%20$0.66
Kev (local)1184 of 118487.2%85.1%89.9%86.5%85.8%84.5%87.2%90.5%20$0.31
Rejecting everything88.5%81.8%88.5%85.8%83.8%82.4%84.5%85.1%

Reviewing fewer stories

Review is the expensive part, so we recorded two runs that ask for less of it. The three runs below share a blank start and an early trigger. The first reviewed all 280 stories and scored 96 percent in the final window. The second reviewed 167 of 280 stories, or 60 percent of them, and scored 94 percent in the final window. The third reviewed 113 of 280 stories, or 40 percent of them, and scored 91 percent in the final window.

The second run spared reviewers 113 stories and its final window was 2 points lower. The third spared 167 stories and its final window was 5 points lower. Each is one run and a window of 70 stories is noisy, so we read this as evidence that less review can work, not as a ranking.

A hosted model and a local one

We ran the same 1,186 stories twice, once with Jev, a hosted model, deciding and once with Kev, a small model that runs locally. Jev scored 95 percent across all 8 windows and cost about 66 cents in model calls at list prices. Kev scored 87 percent across all 8 windows and cost about 31 cents in model calls at list prices. Kev runs on a local machine with no per-call charge, so its cost is the optimizer alone. Rejecting everything would have scored 85 percent.

Jev found 145 of the 177 stories worth publishing, 82 percent, and missed 32. It also published 30 that should have been rejected. Kev found 69 of the 177 stories worth publishing, 39 percent, and missed 108. It also published 45 that should have been rejected.

Jev gave nearly the same answers with and without the learned model, publishing 38 stories that should have been rejected alone and 30 with it. Kev alone published 303 stories that should have been rejected. The learned model pulls that back to 45, at the cost of missing publishable stories, since it found 69 of 177 instead of 140.

Jev

Final decision
PublishedRejected
Should be published177 stories14581.9% of those to publishPublished: right3218.1% of those to publishRejected: wrong
Should be rejected1,007 stories303.0% of those to rejectPublished: wrong97797.0% of those to rejectRejected: right
Each row is the answer we should have given. Heavy borders and bold numbers mark right answers. Of the stories it published, 82.9% were right. Of the stories to publish, it found 81.9%.
Decision model alone
PublishedRejected
Should be published177 stories15084.7% of those to publishPublished: right2715.3% of those to publishRejected: wrong
Should be rejected1,007 stories383.8% of those to rejectPublished: wrong96996.2% of those to rejectRejected: right
Each row is the answer we should have given. Heavy borders and bold numbers mark right answers. Of the stories it published, 79.8% were right. Of the stories to publish, it found 84.7%.

Kev

Final decision
PublishedRejected
Should be published177 stories6939.0% of those to publishPublished: right10861.0% of those to publishRejected: wrong
Should be rejected1,007 stories454.5% of those to rejectPublished: wrong96295.5% of those to rejectRejected: right
Each row is the answer we should have given. Heavy borders and bold numbers mark right answers. Of the stories it published, 60.5% were right. Of the stories to publish, it found 39.0%.
Decision model alone
PublishedRejected
Should be published177 stories14079.1% of those to publishPublished: right3720.9% of those to publishRejected: wrong
Should be rejected1,007 stories30330.1% of those to rejectPublished: wrong70469.9% of those to rejectRejected: right
Each row is the answer we should have given. Heavy borders and bold numbers mark right answers. Of the stories it published, 31.6% were right. Of the stories to publish, it found 79.1%.

A larger set of stories

We also recorded a set of 853 stories, in 6 windows of about 142 stories. Accuracy was 87 percent in the first window and 96 percent in the last. It stayed above the score for rejecting everything in every window. The run cost about 64 cents in model calls at list prices, with 20 optimizer requests.

Where the settings matter

  • Blank start or a written rubric

    A blank start spends its early stories without a rubric. In our run the first window scored 44 percent, well under the 84 percent that always rejecting would have scored. A written start fixes the first window, and it also carries your assumptions into the rubric. A rubric that is too strict makes the decision model reject stories your editors want.

  • When the first rubric may arrive

    The trigger counts label transitions, not labels. When most stories are rejects, transitions are rare. With the default trigger the first rubric came at story 82. With the early trigger it came at story 21. Left at the default, the loop runs the whole first window with no rubric.

  • Where the labels come from

    Our first labels came from one labeling model that over-published. An audit corrected 5 of 10 sampled publish labels and 0 of 10 reject labels. The loop would have learned the over-publishing as if it were the standard. We relabeled before recording.

  • What the decision model sees

    If the reviewer read the title and the decision model did not, accuracy has a ceiling that no rubric can lift. The loop then spends its optimizer calls teaching the decision model things it cannot see.

  • The baseline you compare against

    Publish is the rare label here. A cyclotron that rejects almost everything scores high accuracy and never finds a story to run. Every window on the recording pages sits next to the most-common-label score for that reason.

Risks to watch

  • One run per setting. The numbers show what happened once, not what will happen on your stream.
  • The reviewer is simulated. We have not measured this on real newsroom labels.
  • The relabeled set has few stories, and a window holds only 11 to 15 publish stories, so one story moves publish recall by 7 to 9 points.
  • A fixed confidence threshold set once does not hold its promise as the loop runs on. Re-check it against a random audit.

How this gets set up

The first setup here took decisions on the label policy, the labelers, what the decision model sees, the seed set, the starting rubric, the trigger, the objective, the models, the spending caps and the review rate. Each one has a way to fail quietly. The loop does not tell you which one you got wrong. It steers itself toward whatever you set.

The full list, with the failure each one caused for us, is in the setup guide.

Recordings of this use case

Each recording is one recorded run. Read the setup before comparing.

Featured · on the home page

Blank start: the first rubric lifts accuracy

A cyclotron starts with no written rubric and learns one from reviewer labels as 280 stories arrive. The first rubric proposal waits for 20 label transitions, so it arrives after about 80 stories.

Setup of Blank start: the first rubric lifts accuracy
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 20 cycles
First rubricCycle 82
0%25%50%75%100%70140210280stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.06 in model calls at list prices: 738 decision-model calls ($0.04) and 3 optimizer requests ($0.02).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric; an offline audit found it over-published against that rubric. These v3 labels come from two independent blind relabeling passes under an explicit written policy; only stories where both passes agreed are included.

Kev as the decision model on 1,186 stories

The same study with Kev, a small model that runs locally at no per-call cost, deciding instead of Jev. The Kev revision is pinned.

Setup of Kev as the decision model on 1,186 stories
Starting rubricBlank (no written rubric)
Decision modelKev
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 6 cycles
First rubricCycle 35
0%25%50%75%100%14829644459274088810361184stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.31 in model calls at list prices: 6,607 decision-model calls ($0.00) and 20 optimizer requests ($0.31).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric. These labels come from two independent blind relabeling passes under an explicit written policy; where the passes disagreed (about 3 percent of stories), the label was settled by hand.

Blank start on 853 stories

Cyclotron starts with no rubric and learns one while deciding 853 stories in order, with every label revealed after each decision.

Setup of Blank start on 853 stories
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 6 cycles
First rubricCycle 24
0%25%50%75%100%142284426568710852stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.64 in model calls at list prices: 5,477 decision-model calls ($0.38) and 20 optimizer requests ($0.26).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric. These labels come from two independent blind relabeling passes under an explicit written policy; where the passes disagreed (6 percent of the new stories), the label was settled by hand.

Blank start, first rubric after 21 stories

Same setup as the first recording, but the first rubric is proposed after 6 label transitions instead of 20. The loop writes its first rubric at story 21 and gets about 13 optimizer calls instead of 4.

Setup of Blank start, first rubric after 21 stories
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 6 cycles
First rubricCycle 21
0%25%50%75%100%70140210280stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.25 in model calls at list prices: 2,313 decision-model calls ($0.15) and 13 optimizer requests ($0.10).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric; an offline audit found it over-published against that rubric. These v3 labels come from two independent blind relabeling passes under an explicit written policy; only stories where both passes agreed are included.

Review everything first, then half

Same blank start and early trigger as the full-review run, but after the first 50 stories only half of the stories are reviewed.

Setup of Review everything first, then half
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewed167 of 280 stories (every story for the first 50 stories, then half of the stories)
Rubric changes consideredEvery 6 cycles
First rubricCycle 21
0%25%50%75%100%70140210280stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.08 in model calls at list prices: 904 decision-model calls ($0.04) and 6 optimizer requests ($0.03).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric; an offline audit found it over-published against that rubric. These v3 labels come from two independent blind relabeling passes under an explicit written policy; only stories where both passes agreed are included.

Review less, starting with the rejects

Same blank start and early trigger, but the review schedule is lighter: for the first 50 stories every story the model calls publish and half of those it calls reject are reviewed, then half of the publishes and a quarter of the rejects.

Setup of Review less, starting with the rejects
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewed113 of 280 stories (for the first 50 stories, every story the model calls publish and half of those it calls reject; after that, half of the publishes and a quarter of the rejects)
Rubric changes consideredEvery 6 cycles
First rubricCycle 21
0%25%50%75%100%70140210280stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.05 in model calls at list prices: 637 decision-model calls ($0.03) and 5 optimizer requests ($0.02).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric; an offline audit found it over-published against that rubric. These v3 labels come from two independent blind relabeling passes under an explicit written policy; only stories where both passes agreed are included.

Starting from a written rubric

Instead of a blank start, the cyclotron begins with a written rubric describing what counts as publishable, and the first rubric proposal comes after 6 label transitions.

Setup of Starting from a written rubric
Starting rubricCurated by hand
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 6 cycles
First rubric changeCycle 121
0%25%50%75%100%70140210280stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.22 in model calls at list prices: 2,317 decision-model calls ($0.12) and 13 optimizer requests ($0.10).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric; an offline audit found it over-published against that rubric. These v3 labels come from two independent blind relabeling passes under an explicit written policy; only stories where both passes agreed are included.

Jev as the decision model on 1,186 stories

Cyclotron with Jev deciding, starting from no rubric, learning one over 1,186 stories with every label revealed after each decision.

Setup of Jev as the decision model on 1,186 stories
Starting rubricBlank (no written rubric)
Decision modelJev 1.13.0
LLM optimizergpt-6-luna
ObjectiveAccuracy
Stories reviewedEvery story
Rubric changes consideredEvery 6 cycles
First rubricCycle 35
0%25%50%75%100%14829644459274088810361184stories, to the end of each window
CyclotronDecision model aloneRejecting everything

About $0.66 in model calls at list prices: 5,365 decision-model calls ($0.36) and 20 optimizer requests ($0.30).

Where the labels came from: Labels are simulated. Original labels came from a single labeling model run under a written rubric. These labels come from two independent blind relabeling passes under an explicit written policy; where the passes disagreed (about 3 percent of stories), the label was settled by hand.

Keep reading