Recorded run
Jev as the decision model on 1,186 stories
Cyclotron with Jev deciding, starting from no rubric, learning one over 1,186 stories with every label revealed after each decision.
What to notice
- The cyclotron scored 86.5%, 96.6%, 94.6%, 94.6%, 94.6%, 98.6%, 94.6%, 98.0% across the 8 windows of 148 stories.
- Rejecting everything would have scored 88.5%, 81.8%, 88.5%, 85.8%, 83.8%, 82.4%, 84.5%, 85.1%. Read each accuracy beside that number.
- In the last window the cyclotron beat that baseline by 12.9 points.
- The first rubric proposal arrived at cycle 35. Until then the decision model worked from the blank start.
- In the last window the decision model alone scored 98.0%, so the cyclotron's added work did not help there.
- Jev is the hosted decision model. The first window, before any rubric exists, falls just below the most-common-label baseline; every later window is above it and at 94.6 percent or higher.
- Both studies use the same stories, order, review schedule and optimizer; only the decision model differs.
Setup
| Decision model | Jev 1.13.0 (jev) |
|---|---|
| LLM optimizer | gpt-6-luna |
| Objective | Accuracy |
| Starting rubric | Blank (no written rubric) |
| Stories reviewed | Every story |
| Rubric changes considered | Every 6 cycles |
| Idea screening | Off |
| Max rubric optimizations | null |
| Rubric trigger basis | label_transitions |
| Selection policy | {"aggregation":"positive","max_secondary_regression":0,"minimum_secondary":null,"positive_class":null,"primary":"accuracy","secondary":null} |
| First rubric | Cycle 35 |
| Corpus | editorial-relabel-v5, 1,186 stories |
| Window size | 148 stories |
| Recorded | 2026-10-11 |
| Engine commit | f99dadea86e52144b627e0fdcdf54de2a46a358c |
Accuracy by window
Each window is 148 stories. The last column is what rejecting everything would have scored.
| Window | Cyclotron | Decision model alone | Rejecting everything |
|---|---|---|---|
| Stories 1–148 | 86.5% | 85.8% | 88.5% |
| Stories 149–296 | 96.6% | 95.9% | 81.8% |
| Stories 297–444 | 94.6% | 93.9% | 88.5% |
| Stories 445–592 | 94.6% | 94.6% | 85.8% |
| Stories 593–740 | 94.6% | 94.6% | 83.8% |
| Stories 741–888 | 98.6% | 98.6% | 82.4% |
| Stories 889–1036 | 94.6% | 94.6% | 84.5% |
| Stories 1037–1184 | 98.0% | 98.0% | 85.1% |
How the answers were wrong
Accuracy says how often an answer was wrong. These tables say which way. They count the 1,184 stories whose right answer was revealed.
| Published | Rejected | |
|---|---|---|
| Should be published177 stories | 14581.9% of those to publishPublished: right | 3218.1% of those to publishRejected: wrong |
| Should be rejected1,007 stories | 303.0% of those to rejectPublished: wrong | 97797.0% of those to rejectRejected: right |
| Published | Rejected | |
|---|---|---|
| Should be published177 stories | 15084.7% of those to publishPublished: right | 2715.3% of those to publishRejected: wrong |
| Should be rejected1,007 stories | 383.8% of those to rejectPublished: wrong | 96996.2% of those to rejectRejected: right |
Of 177 stories that should be published, the final decision found 145 and the decision model alone found 150. Of 1,007 stories that should be rejected, the final decision published 30 and the decision model alone published 38.
Cost
About $0.66 in model calls at list prices: 5,365 decision-model calls ($0.36) and 20 optimizer requests ($0.30).
Price basis: list prices, USD per million tokens, applied to the provider-reported usage on every recorded call; decision model jev (TypeSafe's published price); optimizer gpt-6-luna (LiteLLM model price table (standard tier), read 2026-10-11).
Where the labels came from
Labels are simulated. Original labels came from a single labeling model run under a written rubric. These labels come from two independent blind relabeling passes under an explicit written policy; where the passes disagreed (about 3 percent of stories), the label was settled by hand.
Replay the run
One correction can improve the next tested decision.
Cycle 1184 of 1184 · 1184 revealed · recorded agreement
A standalone model repeats its answer. cyclotron records the correction, tests guidance and configuration changes, then uses only accepted evidence to align later decisions.
This recorded decision
Podolski announces retirement from international football
Model: reject · 89%
Reviewer: reject
A player retirement is routine sports news.
Recorded feedback policy: All. Realized feedback is shown in the playback.
Recorded editorial run · Run
Prediction, before the label
reject · 89% confidence
Reviewer label
reject · prediction matched
What the revealed feedback says so far
- Recall
- 90%
- Of the items that should be a yes, how many it found.
- 28 of 31 · 95% interval 75%–97%
- Precision
- 90%
- Of the items it said yes to, how many were right.
- 28 of 31 flagged
- Accuracy
- 97%
- Of all items, how many it got right.
- 194 of 200; always “reject” would score 85%
Accuracy is the share of decisions that matched revealed labels. Precision asks how often a “publish” call was right; recall asks how many actual “publish” stories were found. These are running totals, not future results.
Simulated reviewer replay
Visual replay only — clicking these controls never changes the recorded run.
Editorial decision
Recorded agreement
Recorded simulated reviewer explanation, replayed after the prediction. Ground-truth source evidence remains separate from optimizer feedback. The run records 1184 explanations; the last optimizer proposal came at cycle 529, and the explanations after it were recorded but followed by no optimizer request. Only recorded correction events appear below.
Curated public source
Podolski announces retirement from international football
Wikinews reports: Podolski announces retirement from international football. The original linked article provides the complete reporting context.
Brief adapted and shortened for this demo; see the original Wikinews article for the complete work. Read original · CC BY 2.5 (Wikinews policy; articles dated through 2024-12-15).
What happened after the label
No optimization was due. cyclotron recorded the prediction and the label.
Recorded reliability
Observed accuracy versus the logged pre-review confidence, using only 1184 revealed labels. Hidden labels are not used.
Recorded rolling metrics at this cycle: Brier 0.061 · ECE 0.060.
Matched evaluator-only windows
First 148 vs last 148 items, each scored against frozen simulated reference labels, evaluator-only; never learner feedback.
Accuracy 86% → 98% (rejecting everything would score 89% → 85%) · Brier 0.167 → 0.052 · ECE 0.049 → 0.066.
This permits equal-window comparison across modes; it was not learner-visible feedback.