All guides

Set up

Setting up a cyclotron: the decisions that come first

This is the short version of our setup guide. It lists the decisions that come before the first run and the concrete failure each one caused in our runs. The numbers come from one editorial task, publish or reject a news story, and from recorded runs you can replay. Where we have not measured something, we say so.

Last reviewed:

What the loop does

A cyclotron makes one kind of decision about one kind of item, again and again. A decision model answers each item with a decision and a confidence. A small ML model is fitted on those answers and the reviewers’ labels. Reviewers check some decisions, and a label that differs from the decision is a correction. An LLM optimizer proposes changes, mainly to the rubric, and code checks each proposal before it is promoted or dropped. The corrections are the only signal. If they are wrong or inconsistent, the loop learns that.

Define the decision and the labels

Write the decision as one question a reviewer can answer. Name the labels, and name the positive class. The choices here look small and are not.

  • On a lopsided task, accuracy rewards a cyclotron that gives the common answer every time. In our task 17.8 percent of stories are publish, and always answering reject scores 79 to 84 percent depending on the window. A high accuracy number can hide a decision that never finds the rare class. Compute that baseline before you run anything and report every result next to it.
  • If the reviewer used the story title and the decision model never saw it, accuracy has a ceiling that no rubric can lift. In one test the title lifted a weak starting rubric by about 3 points and did nothing measurable for the strong one. Show the decision model what the reviewer used, then check on a sample whether it mattered.
  • Where the rubric goes in the request matters. For Jev, the same rubric on the same 200 stories scored 77.5 percent when sent in the state and 84.5 percent when sent in the criteria of the decision question. A rubric in the wrong place is partly ignored, and the loop then blames the text.

Where the labels come from, and how consistent they are

The loop optimizes whatever the labels say. Our first editorial labels came from one labeling model that over-published against its own written rubric. An audit corrected 5 of 10 sampled publish labels and 0 of 10 reject labels. Accuracy was capped near 85 percent until the labels were fixed.

We relabeled. Three quick passes agreed only weakly and gave boilerplate reasons. Two careful blind passes under a written policy agreed on 291 of 301 stories. We left the 10 splits out of the corpus instead of settling them by coin flip. One policy line, that an action actually taken counts and a promise or plan does not, decided the corpus.

Low agreement is a sign that the question is not yet well defined. It is not a sign that the labelers are bad. Fix the policy, not the labelers.

The starting point: blank or a written rubric

On the relabeled corpus, Jev with no rubric scored 51.2 percent, because it publishes 65 percent of the stories by default. With our curated starting rubric it scored 90.0. With the written policy text itself as the rubric it scored 87.3. Always rejecting scored 81.1. Written policy is not the same as a good rubric.

We recorded three runs on this corpus. Each is one recorded run. A blank start with the default trigger scored 44 percent in its first window. A blank start with an early trigger scored 77. A written starting rubric scored 91. The final windows were within a few points of each other, and a window is noisy, so we do not rank the three after the first window.

A blank start costs the early stories and keeps your assumptions out of the rubric. A written start gives a good first window and carries your assumptions and biases in with it. On the older labels our curated rubric made Jev say publish for 17 to 20 percent of stories when the editors wanted 40.

The settings that matter

The rubric trigger counts label transitions: consecutive labeled stories whose labels differ. It does not count labels. On a lopsided task transitions are rare, so a trigger of 20 means the first rubric arrives late. In the default-trigger recording the first rubric arrived at story 82. Moving the trigger from 20 to 6 moved it to story 21, and the first window went from 44 percent in the first recording to 77 in the second.

We also ran two batches of six runs each, on the same corpus, one batch at each trigger. These are batches of six runs on one corpus, not a general rule.

  • First rubric at story 82 in all six runs at trigger 20, and at story 21 in all six at trigger 6.
  • First window accuracy 41.4 to 47.1 percent at trigger 20 and 72.9 to 77.1 at trigger 6.
  • Final window median 92.1 percent at trigger 20 and 95.7 at trigger 6.
  • Optimizer requests per run: 4 at trigger 20 and 13 at trigger 6. We have not measured triggers below 6, or either trigger on a balanced task.
  • The objective is accuracy unless one error clearly costs more. In our tests a different objective did not lift accuracy, and at this scale the windows were too small to tell objectives apart.
  • Set a cap on decision calls and on optimizer calls before any paid run, and do a dry run first. A cap that stops a run is an interruption, not a result. We set one cap too low and a run stopped at story 1,031 of 1,200.

The review rate and the thresholds

In one 1,200-story run with a 25 percent review target, a mix of 20 percent least-confident plus a 5 percent random audit had the highest whole-system accuracy at 95.1 percent and the fewest mistakes slipped, at about a sixth of the cost of reviewing everything. Random 25 percent scored 89.6. That was one run per arm, with about 5 points of spread between windows.

A metric-based taper almost never let go. It used 91 percent of the labels, because its calibration bar was about the size of the noise. A time-based taper used 56 percent of the labels and reached 94.3 percent. We have not tested a looser bar.

A fixed threshold drifts. In a long run the mean confidence rose from 0.80 to 0.84 at the same accuracy, so a rule that approves at 0.80 approved more items at lower precision. Whole-system accuracy was 96 to 97 percent in cycles 401 to 600 and 84 percent in cycles 1,001 to 1,100. Keep a random audit running for as long as a threshold is in use, and re-check the threshold against it on a schedule.

What the loop cannot decide for you

The loop optimizes whatever the labels and the objective say, including their biases. Decisions in regulated areas need a documented, auditable setup and monitoring of errors by group. This guide does not say more than that.

  • Who the reviewers are, and whether their corrections are consistent with each other and over time.
  • What “right” means for your decision. Write it down.
  • Which errors cost more. A missed positive and a wrongful positive are rarely the same cost.
  • How errors differ by group. The loop reports overall accuracy and does not compare groups unless you ask it to.
  • What must be logged for audit: the version of the rubric and ML model at each decision, the label, the reviewer, and the explanation.
  • When to stop trusting a threshold.

A setup checklist

In the order we would do it. The full list is longer.

  • Write the decision as one question a reviewer can answer.
  • Name the labels and the positive class.
  • Write down what “right” means, in sentences.
  • Write down which error costs more, and by how much.
  • List who the reviewers will be, and decide what must be logged for audit.
  • Decide how errors will be compared across groups, and where that data comes from.
  • Compute the share of each class and the majority-class baseline.
  • Make what the decision model sees match what the reviewer sees.
  • Write the labeling policy as short rules with a line for each edge case.
  • Label a sample twice, independently, and measure agreement. If it is low, fix the policy and repeat.
  • Hand-audit a sample of each class, and leave out items the labelers split on.
  • Build a seed set with at least 6 labels per class.
  • Check where the rubric goes for your decision model.
  • Choose a starting rubric, blank or written, and record why.
  • Set the rubric trigger low on a lopsided task, and read the first-rubric story.
  • Set caps, then do a dry run and confirm nothing was spent.
  • Choose the review program, and set a date to re-check any fixed confidence threshold against a random audit.
  • Read the rubrics the loop wrote and check them against your policy.
  • Write down what you did not measure.

Supporting reading

Inspect the research and technical explanation.

Next

Choose a practical next step.

Continue learning

The other practical guides.

Understand

What Cyclotron does
See how reviewer corrections can help a decision system learn your team’s standards, and decide whether a task is a good fit.
Read the guide