Bias in automated decisions
Does the answer to a decision about a person move when one personal detail about that person changes?
How we know: Shown by our public benchmark sites. They test models, not a cyclotron.
Where the settings matter
The labels and the objective
The loop optimizes whatever the labels and the objective say, including their biases. Biased labels are learned as if they were true.
Comparing errors by group
The loop reports overall accuracy. It does not compare groups unless you ask it to, so a gap between groups can sit behind a good overall number.
Testing the whole workflow
Accuracy and calibration do not establish fairness. A component can pass a paired test while the final selection outcome gets worse.
What gets logged
Without the version of the rubric and ML model at each decision, the label, the reviewer and the explanation, you cannot investigate a complaint after the fact.
Risks to watch
- In regulated settings, the loop optimizes whatever the labels and objective say, including their biases. Such decisions need a documented, auditable setup and monitoring of errors by group.
- A benchmark measures a model on its test. It does not tell you how your own decision, population and threshold behave. We have not measured that here.
How this gets set up
Our two public benchmarks are linked below. Biased-Decisions measures whether a model's answer moves when one personal detail about a person changes. Hard-Decisions measures how accuracy falls as a decision needs deeper reasoning.
We do not publish a cyclotron result on bias.