What Is LLM-as-a-Judge? How to Check AI Work With a Second Model

LLM-as-a-judge is a way to check AI work with more AI. You give one model’s answer to a second model, along with a written description of what a good answer looks like, and ask the second model to grade it. The judge sends back a score, a pass or fail, or a pick between two answers, usually with a short explanation.
Why should you care? If a chatbot drafts something that matters to you, like your kid’s science fair summary or an email to your boss, you need some way to check it, and a second model is a check you can run yourself. Evaluation tools from Langfuse, LangChain, MLflow and OpenAI all document judge features. The catch is that judges have documented blind spots. Below: how they work, where they fail, and a checklist for using one without getting fooled.
A June 2023 arXiv paper, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” by Lianmin Zheng and colleagues, put the idea to the test. Its headline result: when ties were left out, GPT-4’s verdicts agreed with expert human labelers 85% of the time, while the humans agreed with each other 81% of the time (agreement rate on MT-bench, Table 5, setup S2; run by Zheng et al., reported June 2023). That is the reason everyone got excited. The same paper spends pages on the ways judges fail, which is the reason you should keep reading.
How an LLM judge works
Zheng et al. describe three basic setups, and OpenAI’s evaluation guide lists the same three.
- Single-answer grading. The judge sees one answer and gives it a score. It scales well, but absolute scores “are likely to fluctuate more than relative pairwise results if the judge model changes.”
- Pairwise comparison. The judge sees two answers to the same question and picks the better one, or calls a tie. LangChain’s LangSmith docs make the practical case: picking the more informative of two summaries “is often easier than assigning an absolute score to a single summary.”
- Reference-guided grading. The judge also gets a known-good answer to compare against. It matters a lot for math, as you will see below.
Whichever setup you pick, the judge runs on a rubric: the written definition of good. Langfuse’s docs give a plain example: “Score 1 if the answer is factually incorrect, 5 if fully accurate and well-sourced.”
How you word the rubric moves the results. The G-Eval paper (Yang Liu and colleagues, arXiv March 2023) had the judge write out its evaluation steps first and then fill in a scoring form. With GPT-4 underneath, G-Eval reached a Spearman correlation of 0.514 with human ratings on summaries, averaged across quality dimensions (SummEval benchmark, Table 1; run by Liu et al., reported March 2023). A score of 1 would mean the judge ranked the summaries exactly as the humans did, so 0.514 is partial agreement, though the paper says it beat earlier automatic methods “by a large margin.”
Where judges go wrong
Position bias. Judges tend to favor whichever answer they read first. Zheng et al. gave judges pairs of very similar answers and then swapped the order. GPT-4 gave the same verdict both ways 65.0% of the time with the default prompt, and Claude-v1 only 23.8% of the time (consistency rate, Table 2; run by Zheng et al., reported June 2023). A separate paper, “Large Language Models are not Fair Evaluators” (Peiyi Wang and colleagues, arXiv May 2023), found that just reordering the answers meant “Vicuna-13B could beat ChatGPT on 66 over 80 tested queries with ChatGPT as an evaluator.”
Verbosity bias. Judges like long answers. Zheng et al. padded 23 list-style answers by adding a reworded copy of the same list, with no new information, then asked judges whether the padded version was better. Claude-v1 and GPT-3.5 each fell for it 91.3% of the time, GPT-4 8.7% of the time (failure rate under the “repetitive list” attack, Table 3; run by Zheng et al., reported June 2023). The AlpacaEval team (Yann Dubois and colleagues, arXiv April 2024) built a length-controlled version of their automatic benchmark to correct for this, and report it raised the benchmark’s Spearman correlation with Chatbot Arena rankings from 0.94 to 0.98 (run by Dubois et al., reported April 2024).
Self-preference. Judges may favor their own writing. Zheng et al. saw GPT-4 favor itself with a 10% higher win rate and Claude-v1 with a 25% higher win rate than human judges gave them (difference in win rate excluding ties, Figure 3(b); run by Zheng et al., reported June 2023), though the same authors say that with their limited data they “cannot determine whether the models exhibit a self-enhancement bias.” Later work went further. Arjun Panickssery, Samuel R. Bowman and Shi Feng (arXiv April 2024) found that GPT-3.5 Turbo, GPT-4 and Llama 2 “disproportionately favor summaries written by themselves,” and that GPT-4 could pick out its own summaries from those of two other models and humans with 73.5% accuracy (self-recognition accuracy without fine-tuning; run by Panickssery et al., reported April 2024). G-Eval’s authors saw a related pattern: their GPT-4 judge “always gives higher scores to GPT-3.5 summaries than human-written summaries, even when human judges prefer human-written summaries.”
Getting fooled on math and logic. A judge that can solve a problem on its own can still be talked out of the right answer by the answer it is grading. In Zheng et al.’s test of 10 math questions, each judged in both answer orders, GPT-4 wrongly accepted an incorrect answer in 14 of 20 judgments with the default prompt, 6 of 20 when asked to solve the problem first, and 3 of 20 when given its own solution as a reference (failure count, Table 4; run by Zheng et al., reported June 2023).
The judge is just a prompt. Langfuse’s docs say it well: “A judge is a prompt you wrote, so check that it scores cases the way your reviewers would before you rely on it.”
How to check AI work with a second model: a checklist
Whether you use two browser tabs or a full evaluation tool, the steps are the same.
- Write the rubric before you look at the answer. Use pass or fail, or a small scale such as 1 to 4, and describe what each score means. OpenAI’s guide recommends pass or fail or pairwise comparison “for more reliability.”
- Give the judge a reference when you have one. A known-good answer, the original source document, or the actual numbers. This was the biggest fix in the math test above.
- Ask for reasoning first, score second. G-Eval has the judge write out its evaluation steps before it scores.
- Use a different model than the one that wrote the answer. Self-preference is documented, so do not let a model mark its own homework.
- Swap the order when you compare two answers. Zheng et al.’s conservative rule is to ask twice with the answers swapped and only count a win if the same answer wins both times. Otherwise, call it a tie.
- Watch for length. If the judge keeps picking the longer answer, ask whether it actually says more.
- Test the judge against people. Hand-grade a small set of answers yourself and see how often the judge matches you. OpenAI’s guide says to “validate agreement against your human labels,” and Langfuse suggests measuring judge-to-human agreement with Cohen’s Kappa and a confusion matrix.
- Keep spot-checking after it looks good. Rubrics drift, models get updated, and your inputs change.
Tool options, from the vendors’ own docs (features as described by each vendor):
- Langfuse: LLM-as-a-judge evaluators that run on experiments during development or on individual observations in live traffic, with numeric, categorical or boolean scores.
- LangSmith (LangChain): an LLM-as-a-judge evaluator you can set up in the UI or the SDK, starter templates, and pairwise evaluators. The docs note that few-shot evaluators, which include graded examples in the prompt, “often improve performance.”
- MLflow: custom judges built with
make_judge()(the docs say it requires MLflow 3.4.0 or later), judges that read a full agent trace, and a “judge alignment” feature that tunes a judge against human feedback. - Hugging Face: a step-by-step Hugging Face cookbook notebook (first published March 2024, per its GitHub history) on building a judge prompt and testing it.
- OpenAI Evals: “score model” graders that prompt a separate model to return a numeric score, alongside string-check graders and Python graders for answers that a simple rule or a bit of code can check.
When not to trust a judge
- When nobody knows the right answer. A judge without a reference is grading on vibes plus a rubric. That can be fine for tone and badly wrong for facts.
- When the stakes are high. Medical, legal, money and safety calls get a human, full stop. OpenAI’s own guide admits “No strategy is perfect.”
- When the answers are close. Zheng et al. found GPT-4’s agreement with humans climbed “from 70% to nearly 100%” as the quality gap between the two models grew (agreement rate on non-tied votes, Figure 2; run by Zheng et al., reported June 2023). Their appendix also reports position bias was more noticeable for models with close performance. Judges are weakest exactly where you most need a tiebreaker.
- When the work has many steps. Grading only the final answer of a coding agent misses how it got there. The “Agent-as-a-Judge” paper (Mingchen Zhuge and colleagues, arXiv October 2024) uses an agent to check the steps too, and its authors claim it “dramatically outperforms LLM-as-a-Judge” on their own DevAI benchmark of 55 development tasks. That is the authors’ claim on their own benchmark, not an independent result.
The research also points to a sensible middle path: let the judge handle the easy calls and send the hard ones to a person. Wang et al. built exactly that, using the judge’s own inconsistency to decide which examples needed human review, and report that “with only a 20% human annotation cost” their GPT-4 and ChatGPT judges matched or beat average human alignment (share of examples sent to human annotators; run by Wang et al., reported May 2023).
So treat a second model as a fast first reader with known blind spots. Point it at a clear rubric, give it the answer key when you have one, swap the order, and keep a human signing off on anything that matters.
Sources
- Zheng, L. et al. “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.” arXiv:2306.05685, v1 June 9, 2023 (NeurIPS 2023 Datasets and Benchmarks Track). https://arxiv.org/abs/2306.05685 (PDF: https://arxiv.org/pdf/2306.05685)
- Liu, Y. et al. “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment.” arXiv:2303.16634, v1 March 29, 2023. https://arxiv.org/abs/2303.16634
- Wang, P. et al. “Large Language Models are not Fair Evaluators.” arXiv:2305.17926, v1 May 29, 2023. https://arxiv.org/abs/2305.17926
- Panickssery, A., Bowman, S. R., Feng, S. “LLM Evaluators Recognize and Favor Their Own Generations.” arXiv:2404.13076, v1 April 15, 2024. https://arxiv.org/abs/2404.13076
- Dubois, Y. et al. “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators.” arXiv:2404.04475, v1 April 6, 2024 (COLM 2024). https://arxiv.org/abs/2404.04475
- Zhuge, M. et al. “Agent-as-a-Judge: Evaluate Agents with Agents.” arXiv:2410.10934, v1 October 14, 2024. https://arxiv.org/abs/2410.10934
- Langfuse docs, “LLM-as-a-Judge.” https://langfuse.com/docs/evaluation/evaluation-methods/llm-as-a-judge
- LangChain docs, “How to define an LLM-as-a-judge evaluator” (LangSmith). https://docs.langchain.com/langsmith/llm-as-judge
- LangChain docs, LangSmith “Evaluation concepts.” https://docs.langchain.com/langsmith/evaluation-concepts
- MLflow docs, “Custom Judges.” https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/custom-judges/
- MLflow docs, “Judge Alignment.” https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/alignment/
- OpenAI API docs, “Graders.” https://developers.openai.com/api/docs/guides/graders
- OpenAI API docs, “Evaluation best practices.” https://developers.openai.com/api/docs/guides/evaluation-best-practices
- Hugging Face Open-Source AI Cookbook, “Using LLM-as-a-judge for an automated and versatile evaluation,” by Aymeric Roucher. https://huggingface.co/learn/cookbook/en/llm_judge (notebook history: https://github.com/huggingface/cookbook/commits/main/notebooks/en/llm_judge.ipynb)
Research figures are dated by when each paper was first posted to arXiv; the papers do not give separate run dates, and every figure was re-checked against the current version of the paper. All pages re-opened Saturday, November 14, 2026, between 8:49 and 8:50 PM CT.
