What is LLM-as-a-Judge?

It is an evaluation methodology where a capable, calibrated LLM is used to evaluate, score, and critique the output of another model (or itself) against a detailed rubric with explicit criteria.

What is 'Sycophancy' in LLM evaluation and how do you combat it?

Sycophancy is the tendency of an LLM to flatter, agree with, or give high scores to text simply because it was generated or requested by the user. It is combated by blind grading, strict negative rubrics, few-shot calibration examples, and chain-of-thought grading.

How does an automated Self-Correction Loop work?

The Generator model creates an initial draft; an Evaluator model (or test suite) critiques it against objective criteria; the Critic provides actionable feedback; the Generator refines the draft. This repeats until the score surpasses a threshold.

Evaluation & Self-Correction Loops (AI Critiquing AI)

The Grading Dilemma: Who Watches the Watchers?

In traditional machine learning, evaluating text output was notoriously difficult. Metrics like BLEU and ROUGE simply counted overlapping n-grams. If a candidate wrote "The weather is delightful" and the reference said "It is a lovely sunny day", BLEU scored it near zero despite having identical semantic meaning.

Today, the gold standard for automated quality assurance in complex pipelines is LLM-as-a-Judge, popularized by LMSYS and Zheng et al. (2023). However, if implemented naively, LLM evaluators suffer from severe biases: self-enhancement bias, position bias, verbosity bias, and sycophancy (flattering the input rather than offering rigorous criticism).

The Anatomy of a Calibrated Rubric

Never tell a judge model: "Grade this answer from 1 to 5." A model will give almost everything a 4 or 5. Rigorous evaluation requires an explicit, multi-dimensional rubric with anchor descriptions:

Example 5-Point Accuracy Rubric:
Score 1 (Critical Failure): Contains factually dangerous or completely fabricated claims. Misidentifies core entities.
Score 2 (Deficient): Identifies the correct topic, but makes major factual errors or misses primary user constraints.
Score 3 (Acceptable): Factually sound, but includes unnecessary filler, minor ambiguity, or weak formatting.
Score 4 (Strong): Completely accurate, concise, directly answers the prompt, adheres to all constraints.
Score 5 (Exemplary): Flawless precision, exceptional clarity, anticipates unstated edge cases, zero filler.

The Self-Correction Refinement Loop

When generating high-stakes content (legal briefs, software patches, financial summaries), single-shot generation is rarely sufficient. A closed-loop Critic-Refine cycle elevates output quality significantly:

Generation: Agent A drafts the initial response based on the user prompt.

Critique (Blind Evaluation): Agent B evaluates the draft against the rubric without knowing which model wrote it. It must output an explicit list of flaws:

CRITIQUE:
1. Violated constraint: Exceeded 100-word limit by 24 words.
2. Missing entity: Did not mention the GDPR Article 17 requirement.
3. Tone: Used casual colloquialism in paragraph 2 ("a bunch of stuff").

Refinement: Agent A receives the critique and rewrites the draft, addressing every single numbered item.
Pass/Fail Gate: If score >= 4, the response is delivered to the user; otherwise, the loop allows one final refinement pass.

Combating Evaluator Biases

Position Bias: When comparing Option A vs Option B, models favor Option A up to 60% of the time. Always swap positions and average the scores.
Verbosity Bias: Models naturally correlate longer answers with higher quality. Explicitly penalize verbosity in your scoring instructions.
Require Chain-of-Thought Grading: Never allow the judge model to emit a score first. Force it to write its detailed reasoning before outputting the final numerical rating.

Evaluation & Self-Correction Loops (AI Critiquing AI)

The Grading Dilemma: Who Watches the Watchers?

The Anatomy of a Calibrated Rubric

The Self-Correction Refinement Loop

Combating Evaluator Biases

Topic 39 Scenario Challenge.