Most RAG demos stop at "it answered my question." Here is the small harness that belongs in front of every retrieval system before it ships — and the reason it is three numbers rather than one.
Three numbers, not one
- Retrieval recall tells you whether the right passage was found.
- Faithfulness tells you whether the answer stuck to what was found.
- Correctness tells you whether the user got what they needed.
A single "quality" score hides which of the three broke. Low recall means fix chunking or the index. Low faithfulness with high recall means fix the prompt. Low correctness with both high means the documents themselves are wrong or missing.
The core loop
def run_eval(dataset, pipeline, judge):
results = []
for case in dataset:
hits = pipeline.retrieve(case.question)
answer = pipeline.answer(case.question, hits)
results.append(Score(
recall=recall_at_k(hits, case.gold_ids, k=5),
faithful=judge.faithful(answer, hits),
correct=judge.correct(answer, case.expected),
))
return summarize(results) # fails CI below thresholdrecall_at_k is plain set arithmetic. judge.faithful and judge.correct are LLM-as-judge calls with a fixed rubric and a fixed model, so the judge doesn't drift with the system under test.
Building the test set
Start with 30–50 questions you'd be embarrassed to get wrong. A small set that runs on every pull request beats a big one nobody runs.
Pull questions from real support tickets, search logs and the people who answer them today. For each, record the expected answer and the document passage that supports it — that passage's ID is the "gold" for recall. Add a handful of questions the system should refuse, with no gold passage, to catch confident guessing.
Gating releases on it
The harness runs in CI on every change to the index, the prompt, the retriever or the model. Each number has a threshold set from the first baseline run. A drop below any threshold fails the build, with a table of the cases that regressed so the fix is obvious. Thresholds move up, never down, as the system improves.
What this costs
For a 50-question set with two judge calls per question, a full run is a hundred small LLM calls — a few cents and a couple of minutes. It is the cheapest insurance in the whole system.