Which AI is the best debate judge?

The DebateIt Judge Benchmark runs the flagship models of Anthropic, OpenAI, Google, xAI, and DeepSeek on the same real British Parliamentary out-rounds and scores each model's 1-2-3-4 team ordering against the human chair's actual call. The live leaderboard on this page ranks them by pairwise agreement with the chair.

Can an AI judge a competitive debate round?

DebateIt's AI judge writes a complete ballot with a reason for decision on any pasted round in seconds. On chair-labeled tournament out-rounds, frontier models order teams the same way as the human chair well above random baselines; the exact numbers, sample size, and caveats are published on this page. A second track scoring the judge against expert ratings of platform rounds is being built, and that agreement number publishes only when the rated set exists.

How is agreement measured?

On the tournament track, a BP call ranks four teams, giving six head-to-head pairs; agreement is the share of pairs the model orders the same way as the human chair, alongside same-winner and exact-ordering rates. On the platform track, a round counts as agreement when the AI ballot's winner matches the winner chosen by the expert rater, an APDA Pro-Ams champion (2025), scoring complete rounds speech by speech against format-specific judging criteria.

Can I test the AI judge myself?

Yes. Paste any round transcript at debateai.com/judge and the judge returns a full ballot with a reason for decision in seconds, free.

Judge benchmark

Can AI judge a debate round?

It is a live argument on every circuit. We are building the answer in public, on two tracks: real tournament rounds where a human chair's call is the gold label, and platform rounds rated by an expert. A benchmark you can't interrogate is marketing.

Track one: which lab's model judges best

Real British Parliamentary out-rounds and bubble rounds from tournaments including Vienna IV, Seattle IV, Hart House IV, Yale IV, Columbia IV, the Zagreb and Drexel pre-WUDC opens, Tokyo, Ottawa, Berkeley, and two tournament finals. An experienced human chair sat each one and made the call. Each lab's flagship model gets the same decontaminated judge flow of the round, the same adjudication prompt, and one job: chair it, and order all four teams.

	Lab	Agreement with the chair	Same winner	Exact 1-2-3-4
1	OpenAIgpt-5.2-2025-12-11	79%	77%	41%
2	Anthropicclaude-opus-4-8	74%	67%	33%
3	DeepSeekdeepseek-v4-flash	72%	68%	45%
4	xAIgrok-4.3	70%	45%	27%
	Random baselinecoin-flip judging	50%	25%	4%

Run July 18, 2026 · 22 chair-labeled BP rounds · one flagship model per lab, identical prompts; the model id shown is the one the lab's API reported serving. An API error on a round drops it from that model's average (Anthropic scored 21 of 22). Google's Gemini is not on the board yet: the available API key ran out of quota before a clean pass; the slot is open and gets filled on the next run.

How to read it. A BP call ranks four teams, which gives six head-to-head pairs; "agreement with the chair" is the share of those pairs the model orders the same way the human did, and it is the headline metric because it gives partial credit on close rounds. "Same winner" is whether the model's first place matches the chair's. "Exact 1-2-3-4" is the whole ordering, all four positions, and it is brutal by design.

Before you quote it. The sample is small; single-digit gaps between adjacent models are noise, and the gap over the random baseline is the signal. The inputs are a chair's terse flow notes, not full transcripts, so every score is a noisy lower bound. And the gold label is one good chair, not ground truth from heaven: several of these rounds were split panel decisions, and human judges sitting the same round disagree with each other too. Perfect agreement with any single chair is not an achievable 100%.

The challenge. This page is a standing invitation to AI labs. Judging a debate well means tracking clash across eight speeches, weighing magnitude against probability, making drops cost something, and refusing to reward fluent delivery of a refuted argument. If your model does it better, we will run it and publish the result: same rounds, one identical prompt per round, an unparseable ballot gets one retry and then counts as an error. The gold labels (motions, tournaments, orderings) are shareable; the flows stay private because they name real debaters. Write us.

Track two: the platform track

rounds and speeches logged to the corpus (signed-in rounds only; guest rounds are never stored)

rounds in the published expert-rated set. The number this page exists to grow.

formats, each with its own encoded judging norms

2,195

published judge paradigms distilled into those criteria

As of July 4, 2026. Every count above reconciles to a live database query. It updates as the rated corpus grows.

Track one scores models on tournament rounds that already have a human call. Track two is the harder, slower promise: platform rounds (typed and voice), each fully rated by an expert, with the AI ballot scored against that rating. It is where the benchmark stops depending on any single chair and starts reconciling to a live database.

What is being measured

A round counts as agreement when the AI ballot's winner matches the winner in the expert rating of the same round. Not "did the AI say something plausible": the same transcript, two independent verdicts, do they land on the same side.

The judge does not return a bare verdict. Every ballot is a full reason for decision: which arguments survived, which were dropped, how the weighing resolved, speech-by-speech notes. That is the part debaters actually learn from, and it is also what makes disagreements inspectable: when the AI and the expert split, you can read both rationales and decide who judged the round better.

Methodology

The judge engine. Format-specific adjudication criteria per format (BP, APDA, Policy, LD, PF, WSDC, Asian Parliamentary, Congress, and others), each encoding that format's real judging norms: speaker-scale bands, drop severity, weighing conventions, grounded in 2,195 published judge paradigms from real tournament pools. The engine decides on the flow; instructions that name a winner are dead on arrival.
The expert. Rated rounds are scored by an APDA Pro-Ams champion (2025), speech by speech, against the same format's criteria.
The set. Complete rounds run on the platform (typed and voice) that receive a full expert rating. Today that set is empty. Building it is the current work, and the count above is queryable, not estimated.
The metric. Winner agreement: AI ballot winner = expert-rated winner. The agreement number publishes when the set exists, and not before.

Where this stands, honestly

The rated set is empty today. An earlier version of this page reported estimated figures, including an estimated agreement rate. Track two now reports only counts that reconcile to a live database query, and there is no platform agreement number yet; the only published numbers are track one's, from the tournament harness above.
One rater at first. Panel judging exists precisely because experts disagree; human judge-to-judge agreement in competitive debate is itself well below 100%, which is important context for reading any number that will ever appear on this page. Additional independent raters are part of the plan.
Selection effects. Rated rounds will be platform rounds, not tournament finals. Different distribution of skill and formats than the open circuit.

All three shrink the same way: more rated rounds, more raters. The numbers on this page will move as that happens, in either direction.

Test it yourself

The benchmark you can run right now beats the one you read about. Paste any round transcript and get the full ballot:

Get an AI ballot on any round →

Citing this. "DebateIt AI Judge Benchmark, July 2026: lab leaderboard published on 22 chair-labeled BP out-rounds (small sample, flow-note inputs, scores are a noisy lower bound); platform expert-rated track in progress, no platform agreement number yet." Link this page. Questions or want the methodology in more depth? Write us.