What Is LLM-as-a-Judge? How to Check AI Work With a Second Model

AI

LLM-as-a-judge is a way to check AI work with more AI. You give one model’s answer to a second model, along with a written description of what a good answer looks like, and ask the second model to grade it. The judge sends back a score, a pass or fail, or a pick between two answers, usually with a short explanation.

Why should you care? If a chatbot drafts something that matters to you, like your kid’s science fair summary or an email to your boss, you need some way to check it, and a second model is a check you can run yourself. Evaluation tools from Langfuse, LangChain, MLflow and OpenAI all document judge features. The catch is that judges have documented blind spots. Below: how they work, where they fail, and a checklist for using one without getting fooled.

A June 2023 arXiv paper, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” by Lianmin Zheng and colleagues, put the idea to the test. Its headline result: when ties were left out, GPT-4’s verdicts agreed with expert human labelers 85% of the time, while the humans agreed with each other 81% of the time (agreement rate on MT-bench, Table 5, setup S2; run by Zheng et al., reported June 2023). That is the reason everyone got excited. The same paper spends pages on the ways judges fail, which is the reason you should keep reading.

How an LLM judge works

Zheng et al. describe three basic setups, and OpenAI’s evaluation guide lists the same three.

Whichever setup you pick, the judge runs on a rubric: the written definition of good. Langfuse’s docs give a plain example: “Score 1 if the answer is factually incorrect, 5 if fully accurate and well-sourced.”

How you word the rubric moves the results. The G-Eval paper (Yang Liu and colleagues, arXiv March 2023) had the judge write out its evaluation steps first and then fill in a scoring form. With GPT-4 underneath, G-Eval reached a Spearman correlation of 0.514 with human ratings on summaries, averaged across quality dimensions (SummEval benchmark, Table 1; run by Liu et al., reported March 2023). A score of 1 would mean the judge ranked the summaries exactly as the humans did, so 0.514 is partial agreement, though the paper says it beat earlier automatic methods “by a large margin.”

Where judges go wrong

Position bias. Judges tend to favor whichever answer they read first. Zheng et al. gave judges pairs of very similar answers and then swapped the order. GPT-4 gave the same verdict both ways 65.0% of the time with the default prompt, and Claude-v1 only 23.8% of the time (consistency rate, Table 2; run by Zheng et al., reported June 2023). A separate paper, “Large Language Models are not Fair Evaluators” (Peiyi Wang and colleagues, arXiv May 2023), found that just reordering the answers meant “Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator.”

Verbosity bias. Judges like long answers. Zheng et al. padded 23 list-style answers by adding a reworded copy of the same list, with no new information, then asked judges whether the padded version was better. Claude-v1 and GPT-3.5 each fell for it 91.3% of the time, GPT-4 8.7% of the time (failure rate under the “repetitive list” attack, Table 3; run by Zheng et al., reported June 2023). The AlpacaEval team (Yann Dubois and colleagues, arXiv April 2024) built a length-controlled version of their automatic benchmark to correct for this, and report it raised the benchmark’s Spearman correlation with Chatbot Arena rankings from 0.94 to 0.98 (run by Dubois et al., reported April 2024).

Self-preference. Judges may favor their own writing. Zheng et al. saw GPT-4 favor itself with a 10% higher win rate and Claude-v1 with a 25% higher win rate than human judges gave them (difference in win rate excluding ties, Figure 3(b); run by Zheng et al., reported June 2023), though the same authors say that with their limited data they “cannot determine whether the models exhibit a self-enhancement bias.” Later work went further. Arjun Panickssery, Samuel R. Bowman and Shi Feng (arXiv April 2024) found that GPT-3.5 Turbo, GPT-4 and Llama 2 “disproportionately favor summaries written by themselves,” and that GPT-4 could pick out its own summaries from those of two other models and humans with 73.5% accuracy (self-recognition accuracy without fine-tuning; run by Panickssery et al., reported April 2024). G-Eval’s authors saw a related pattern: their GPT-4 judge “always gives higher scores to GPT-3.5 summaries than human-written summaries, even when human judges prefer human-written summaries.”

Getting fooled on math and logic. A judge that can solve a problem on its own can still be talked out of the right answer by the answer it is grading. In Zheng et al.’s test of 10 math questions, each judged in both answer orders, GPT-4 wrongly accepted an incorrect answer in 14 of 20 judgments with the default prompt, 6 of 20 when asked to solve the problem first, and 3 of 20 when given its own solution as a reference (failure count, Table 4; run by Zheng et al., reported June 2023).

The judge is just a prompt. Langfuse’s docs say it well: “A judge is a prompt you wrote, so check that it scores cases the way your reviewers would before you rely on it.”

How to check AI work with a second model: a checklist

Whether you use two browser tabs or a full evaluation tool, the steps are the same.

  1. Write the rubric before you look at the answer. Use pass or fail, or a small scale such as 1 to 4, and describe what each score means. OpenAI’s guide recommends pass or fail or pairwise comparison “for more reliability.”
  2. Give the judge a reference when you have one. A known-good answer, the original source document, or the actual numbers. This was the biggest fix in the math test above.
  3. Ask for reasoning first, score second. G-Eval has the judge write out its evaluation steps before it scores.
  4. Use a different model than the one that wrote the answer. Self-preference is documented, so do not let a model mark its own homework.
  5. Swap the order when you compare two answers. Zheng et al.’s conservative rule is to ask twice with the answers swapped and only count a win if the same answer wins both times. Otherwise, call it a tie.
  6. Watch for length. If the judge keeps picking the longer answer, ask whether it actually says more.
  7. Test the judge against people. Hand-grade a small set of answers yourself and see how often the judge matches you. OpenAI’s guide says to “validate agreement against your human labels,” and Langfuse suggests measuring judge-to-human agreement with Cohen’s Kappa and a confusion matrix.
  8. Keep spot-checking after it looks good. Rubrics drift, models get updated, and your inputs change.

Tool options, from the vendors’ own docs (features as described by each vendor):

When not to trust a judge

The research also points to a sensible middle path: let the judge handle the easy calls and send the hard ones to a person. Wang et al. built exactly that, using the judge’s own inconsistency to decide which examples needed human review, and report that “with only a 20% human annotation cost” their GPT-4 and ChatGPT judges matched or beat average human alignment (share of examples sent to human annotators; run by Wang et al., reported May 2023).

So treat a second model as a fast first reader with known blind spots. Point it at a clear rubric, give it the answer key when you have one, swap the order, and keep a human signing off on anything that matters.

Sources

  1. Zheng, L. et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” arXiv:2306.05685, v1 June 9, 2023 (NeurIPS 2023 Datasets and Benchmarks Track). https://arxiv.org/abs/2306.05685 (PDF: https://arxiv.org/pdf/2306.05685)
  2. Liu, Y. et al. “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.” arXiv:2303.16634, v1 March 29, 2023. https://arxiv.org/abs/2303.16634
  3. Wang, P. et al. “Large Language Models are not Fair Evaluators.” arXiv:2305.17926, v1 May 29, 2023. https://arxiv.org/abs/2305.17926
  4. Panickssery, A., Bowman, S. R., Feng, S. “LLM Evaluators Recognize and Favor Their Own Generations.” arXiv:2404.13076, v1 April 15, 2024. https://arxiv.org/abs/2404.13076
  5. Dubois, Y. et al. “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators.” arXiv:2404.04475, v1 April 6, 2024 (COLM 2024). https://arxiv.org/abs/2404.04475
  6. Zhuge, M. et al. “Agent-as-a-Judge: Evaluate Agents with Agents.” arXiv:2410.10934, v1 October 14, 2024. https://arxiv.org/abs/2410.10934
  7. Langfuse docs, “LLM-as-a-Judge.” https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge
  8. LangChain docs, “How to define an LLM-as-a-judge evaluator” (LangSmith). https://docs.langchain.com/langsmith/llm-as-judge
  9. LangChain docs, LangSmith “Evaluation concepts.” https://docs.langchain.com/langsmith/evaluation-concepts
  10. MLflow docs, “Custom Judges.” https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/custom-judges/
  11. MLflow docs, “Judge Alignment.” https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/alignment/
  12. OpenAI API docs, “Graders.” https://developers.openai.com/api/docs/guides/graders
  13. OpenAI API docs, “Evaluation best practices.” https://developers.openai.com/api/docs/guides/evaluation-best-practices
  14. Hugging Face Open-Source AI Cookbook, “Using LLM-as-a-judge for an automated and versatile evaluation,” by Aymeric Roucher. https://huggingface.co/learn/cookbook/en/llm_judge (notebook history: https://github.com/huggingface/cookbook/commits/main/notebooks/en/llm_judge.ipynb)

Research figures are dated by when each paper was first posted to arXiv; the papers do not give separate run dates, and every figure was re-checked against the current version of the paper. All pages re-opened Saturday, November 14, 2026, between 8:49 and 8:50 PM CT.