Your LLM Reviewer Agrees With Itself. That's the Bug.

When one LLM generates content and another from the same family reviews it, they can agree completely — and both be wrong in the same direction.

That's not a capability failure. It's family bias: a structurally correlated blind spot that no amount of prompting fixes, because the judge and producer share training patterns — they find each other's outputs "familiar" in a way that operates below the prompt layer. The 2025 research on this is now strong enough to act on.

I built a two-judge review loop — one Anthropic model, one OpenAI model — with a convergence rule that defines what "agreement" actually means. After ~75 review items and dozens of iteration loops, the pattern held: the second judge caught a class of errors the first couldn't, not because it was smarter, but because it wasn't correlated with the producer.

There's also an open-source tool so you can run it against your own content.

Why single-judge fails quietly

Frontier models are competent judges. That's precisely why this failure mode is dangerous — the verdicts look authoritative.

Spiliopoulou et al.'s "Play Favorites" (Aug 2025) showed that GPT-4o and Claude 3.5 Sonnet don't just over-rate their own outputs — they over-rate outputs from their entire model lineage. Wataoka et al. (arXiv 2410.21819, rev. Jun 2025) traced the mechanism to pattern familiarity: judges find text from familiar architectures easier to process and rate it higher. Adaline's April 2026 analysis found frontier judges failing more than 50% of bias stress tests, with the practical takeaway: "Separate your judge's model family from your generator's."

The solution isn't a smarter judge. It's a judge that can't share the producer's blind spots.

The convergence rule

I use two independent judges per item:

• Judge A: Anthropic (Claude Sonnet by default)

• Judge B: OpenAI (GPT-4o by default)

Each sees the same content and criteria. Neither sees the other's output. Each returns:

• confidence — certainty in the verdict, 0–100

• verdict — APPROVED, FLAGGED, or UNCERTAIN

• rationale — one sentence

A decision is trusted when two conditions both hold:

Both are required. Average alone misses divergence; spread alone misses low-confidence agreement.

Examples from actual runs:

My calibration: avg ≥ 75, spread ≤ 15, max 4 iterations. Tune these against your corpus — the shape transfers, the numbers don't.

When a verdict doesn't meet both conditions, I don't discard the item — I escalate.

The escalation ladder

The escalation trigger is specific: if improvement between iterations is ≤ 5 confidence points, more of the same model won't close the gap.

Two things I learned running this:

I learned not to escalate both judges when one was diverging. If Judge A was consistent across iterations and Judge B was all over the place, escalating both burned budget without isolating the signal. The ladder reflects this — it steps up Judge A (Anthropic) first, since in my experience Opus is more reliable than Sonnet on genuinely ambiguous items, while Judge B stays on gpt-4o through L3.

I capped at 4 iterations. After 4 loops without convergence, the item is genuinely ambiguous. The right response is a human decision, not a fifth model call.

See it running

uv run python3 validate.py \
  --content "Cache relies on TTL alone, no explicit eviction path." \
  --criteria "Flag any caching pattern without an eviction or fallback path."

Dual-LLM Validator
  Judge A : claude-sonnet-4-6
  Judge B : gpt-4o
  Rule    : avg >= 75  AND  spread <= 15  |  max 4 iters

Iteration 1 / 4
  Judge A  claude-sonnet-4-6   confidence  95  FLAGGED
  Judge B  gpt-4o              confidence  95  FLAGGED

  avg 95.0  |  spread 0  |  CONVERGED ✅

Result     : FLAGGED
Iterations : 1

Rationales:
  Judge A: TTL-only caching creates silent stale reads
           when upstream data changes without eviction.
  Judge B: No eviction path means stale entries persist
           until TTL expires, regardless of data changes.

When it doesn't converge on the first pass:

Iteration 1 / 4
  Judge A  claude-sonnet-4-6   confidence  85  FLAGGED
  Judge B  gpt-4o              confidence  48  UNCERTAIN

  avg 66.5  |  spread 37  |  NOT CONVERGED (verdicts disagree)

  >> Trust delta stalled — escalating to L2

Iteration 2 / 4  [L2]
  Judge A  claude-opus-4-7     confidence  80  FLAGGED
  Judge B  gpt-4o              confidence  76  FLAGGED

  avg 78.0  |  spread 4  |  CONVERGED ✅

L2 escalates Judge A to Opus (stronger on ambiguous items); Judge B stays on gpt-4o — the divergence was on the Anthropic side.

Try it yourself

git clone https://github.com/deepakgoyal-ai/dual-llm-validator
cd dual-llm-validator
uv sync
cp .env.example .env   # then open .env and fill in your keys

uv run python3 validate.py \
  --content "your content here" \
  --criteria "your review criteria here"

Custom models and thresholds:

uv run python3 validate.py \
  --content-file review.txt \
  --criteria-file criteria.txt \
  --model-a claude-opus-4-7 \
  --model-b gpt-4-turbo \
  --avg-floor 80 \
  --spread-max 10

Exit code 0 = converged. Exit code 2 = human review required. Pipe-friendly.

The operational discipline that matters most

Two things that aren't about the models at all:

I re-validate after every fix. First dispatch + fix applied ≠ done. A verdict is closed only when a re-validation pass converges on the corrected content. Skipping re-validation means I've applied a fix that passed one judge once — which is the original failure mode.

I managed two token budgets, not one. Running two providers concurrently means two rate limits refilling on different schedules. I alternated between providers across dispatch batches rather than parallelizing everything. Parallel looks faster until one provider hits a rate limit 40% through and stalls everything.

What I'm still unsure about

Do these thresholds transfer? I calibrated avg ≥ 75 and spread ≤ 15 against one corpus. There's no reason to expect them to be correct for a different review task. Calibrate against your own items before trusting the defaults.

Did I trade family bias for a different correlated bias? Cross-family reduces family bias — the 2025 papers document this. It doesn't eliminate correlated failure. Two models trained heavily on the same internet text could still share blind spots I haven't characterized.

When shouldn't you do this? If the cost of a wrong verdict is low, single-judge is cheaper and probably fine. If you have labelled data and bandwidth, a fine-tuned single judge likely outperforms two general-purpose ones. This pattern is the 0-to-1 case — no training data, real stakes, need something better than one model family's opinion.

The short version: two judges, different families, convergence = avg ≥ floor AND spread ≤ max, cap at 4 iterations, accept ambiguity rather than escalating forever.

The rest is calibration.

I'm a Technical Architect building AI-enabled systems. Writing about what I actually build — not what I think I should build.

If you're running a review pipeline and have calibrated your own thresholds, I'd like to know what you landed on.