Support-call quality review
Does this call meet the standard the rubric sets?
How we know: From production experience. We did this in production for Call Criteria. It is described here, and you cannot replay it.
Who makes this decision today
Experts review a sample of the calls that classifiers score.
What a correction looks like
Classifiers score calls. An expert reviews a sample and explains each disagreement. The correction becomes a training example, and the explanation becomes a written rule.
Where the settings matter
Where the learning comes from
Historical labels were unreliable, so the lab learned from live corrections instead of a training set. Training on the unreliable labels would have taught the unreliability.
What counts as an explanation
Explanations become written rules. A vague explanation becomes a vague rule.
When a new version goes live
A new version goes live only after held-out review passes. Without that gate, an unchecked change reaches the calls.
Risks to watch
- Setting up a new rubric with AI assistance within an hour is initial setup. Validation and production rollout have their own evaluation and review steps.
- This is described, not replayable. We have not published a recording of it.
- Each decision and workflow needs its own evaluation and review criteria.
How this gets set up
We built this loop for Call Criteria before Cyclotron had a name. The case study tells that history, and the home page quotes what it reports: hundreds of concurrent rubrics and millions of calls scored in production.