Validate the Judge Before Evaluating RAG Retrieval

In a RAG retrieval experiment, the first question was whether a proposed query brought back better search results than the baseline. I built an evaluation pipeline that compared the two result sets with an LLM-as-a-Judge. Then a swap test showed that the judge changed its decision based on answer order in about 81% of cases. The conclusion changed from "which query won" to "the evaluator has to be validated first."

Summary

• Problem: In a real-time financial information RAG system, manually judging retrieval results for every case was not practical.
• Approach: Compare baseline-query and proposed-query results with an LLM judge, then evaluate the same pair in both A/B and B/A order.
• Result: The initial results made the proposed query look stronger, but the swap test exposed about 81% position bias.
• Lesson: LLM-as-a-Judge is an evaluation automation tool, not a source of truth. Retrieval quality evaluation also needs judge stability checks.

Context

During an undergraduate research program, I built a pipeline to evaluate retrieval quality for real-time financial information RAG. The goal was to compare which result set was more useful: one retrieved by a baseline query, or one retrieved by a proposed query.

In RAG, retrieval quality is not just about bringing back more documents. The result has to contain evidence that can answer the question, reflect current financial context, and avoid irrelevant documents. When there is no pre-labeled gold document set, recall and precision alone are not enough.

So I used an LLM as a judge. In this post, "judge" does not mean a fine-tuned model. It means a comparison evaluator controlled by a prompt that defines criteria and output format.

The First Evaluation Pipeline

The first pipeline was simple.

1. Generate a baseline query and a proposed query from the same input.
2. Retrieve search results for each query.
3. Ask the LLM judge to compare the two result sets.
4. Evaluate the same pair once as A/B and once as B/A.
5. Count a winner only when both evaluations point to the same real answer.

My initial expectation was that the proposed query would perform better because it included role assignment, step-by-step reasoning, and financial context in query generation.

The first judge results seemed to support that expectation.

From those numbers alone, it is tempting to say that the proposed query was better. But there was a more important question to answer first.

Is this judge fair?

The Swap Test Changed the Conclusion

When an LLM judge compares two answers, its decision can change depending on which answer is shown first, even if the content is the same. So I evaluated each pair twice.

First pass: A vs B
Second pass: B vs A
If both passes choose the same real answer: consistent
If the winner changes after swapping order: position-biased

In the final presentation material, about 81% of cases showed position bias.

That result changed the center of the experiment. I could no longer separate whether the proposed query was better from whether the judge preferred the answer shown first. While trying to automate evaluation, I first found evidence that the evaluator itself was unstable.

Why This Happened

Position bias becomes easier to see when both results look similarly plausible. When the quality gap between A and B is large, the judge tends to be more stable. But when both look good enough, the judge can be influenced by signals other than actual retrieval quality.

• The answer presented first
• A longer or more structured answer
• Sentences that look more current
• More confident wording
• Small wording differences in the judge prompt

This changed how I looked at LLM-as-a-Judge. It is not a full replacement for human evaluation. It is a cheap and fast evaluation system that helps narrow candidates. If it is a system, its failure modes also need to be tested.

What I Changed

I made the evaluation pipeline more conservative.

This approach does not produce many wins. Instead, it separates cases where the judge was unstable from cases where a decision survived the order swap. The goal of automation changed from "reach conclusions quickly" to "filter out conclusions that are not trustworthy."

Verification

The verification did not only look at query performance. It also checked judge consistency.

• Sent baseline and proposed retrieval results to the judge under the same criteria.
• Evaluated each pair in both A/B and B/A order.
• Checked whether the winner stayed the same in terms of the real answer, not the display slot.
• Marked conflicting cases as review-needed instead of forcing a winner.

This makes the number "proposed query won by X%" less impressive. But the remaining wins have at least passed one position-bias check.

How I Would Judge This Next Time

• When using LLM-as-a-Judge, validate judge stability before looking only at the evaluated target.
• Treat A/B and B/A swap tests as the default for pairwise comparison.
• Do not claim that one query strategy is better from a single judge pass.
• When two results are both reasonably good, keep the case as review-needed instead of forcing a tie or winner.
• The goal of evaluation automation is not to create more conclusions. It is to filter out conclusions that should not be trusted.

References

• Gemini API Fine-tuning documentation
• Gemini API release notes
• Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge