RAG Evaluation Metrics: How to Test a RAG Chatbot Before You Ship It

A RAG chatbot fails quietly. It still answers, in confident prose, even when retrieval has returned the wrong passage or the model has ignored the context. Without a way to measure that, every change to chunk size, embedding model, prompt or provider is a guess. This guide covers how to build an evaluation you can repeat, and how to describe its results without overstating them.

Evaluate retrieval and generation separately

A wrong answer has two possible causes, and they need different fixes. Either the right passage never reached the model (a retrieval problem), or it did and the model used it badly (a generation problem). If you only score the final answer you cannot tell which, so measure both stages.

Retrieval metrics

• Hit rate or context recall. For a question with a known source passage, did that passage appear in the top results? This is the first number to look at, because nothing downstream can recover a passage that was never retrieved.
• Precision. How much of what was retrieved is relevant? Low precision floods the model with noise, which raises cost and tempts it to blend unrelated facts.
• Ranking quality. Measures such as mean reciprocal rank reward putting the right passage first, which matters when you pass only a few passages to the model.

Tune the number of passages and any similarity cut-off against these numbers rather than by feel, and remember that a cut-off that is too strict produces confident refusals, while one that is too loose produces confident nonsense.

Generation metrics

• Groundedness (faithfulness). Is every claim in the answer supported by the retrieved passages? This is the closest measurable proxy for hallucination.
• Answer relevance and completeness. Does the answer address the question that was asked, and does it include the parts that matter?
• Citation correctness. A citation that points at a real retrieved passage is not the same as a citation that supports the sentence it follows. Checking that the reference exists is easy and worth doing, but it only proves the format, not the truth.
• Refusals. When the documents do not contain the answer, the bot should say so. Include questions that cannot be answered and check that it declines.
• Forbidden claims. List statements the bot must never make, such as promising a refund the policy does not allow, and check that they never appear.

Build a test set that earns its name

Thirty to one hundred questions is a useful starting size. Draw them from real sources: support tickets, search logs, questions the client's staff actually ask. Avoid questions you wrote by copying a sentence from a document, because they are trivially easy. For each question record the passage or passages that should be retrieved, and where it helps, the key facts a correct answer must contain. Add unanswerable questions and tricky ones: negations, numbers, questions that span two documents, and wording that does not match the document's vocabulary.

Keep the test set stable. Changing questions after a failure so the numbers recover is the quickest way to make an evaluation meaningless.

Choose checks you can repeat

Two families of check complement each other.

• Deterministic checks are cheap and stable: the expected source was retrieved, a forbidden phrase is absent, every citation tag refers to a retrieved passage, the answer arrived within a time limit. They run on every change and never disagree with themselves.
• Model-graded checks use a second language model as a judge of groundedness or relevance. They catch nuance a string match cannot, but a judge has its own biases and varies between runs. Calibrate it by comparing its verdicts with human labels on a sample, and record which judge model and prompt you used.

Whichever you use, add periodic human review of a random sample. It is the only check that can catch what both automated methods miss.

Make it a regression test

The real value of an evaluation is comparing runs. Freeze and record the whole configuration with each run: chunk size and overlap, embedding model, number of passages, similarity cut-off, system prompt, generation model and its settings. A short fingerprint of that configuration lets you see at a glance whether two results are comparable. Then make a rule that a change to any of those settings does not ship until the suite has run and the differences are understood.

Measure latency and cost per question

Record the time each question took and report percentiles, not just an average. For cost, use the token counts that the provider returns with each response, multiplied by the provider's current prices. A flat per-answer estimate looks tidy in a report but hides the expensive cases.

Reporting results to a client

A pass rate is only meaningful with its definition beside it. A useful report states the test set size and where the questions came from, the configuration that was tested, what each check means, which cases failed and why, and the date. It also states limits. A passing score means the bot did well on those questions with those documents. It does not guarantee behavior on questions nobody wrote, and a citation check shows that an answer cites a source, not that the answer is correct.

Common mistakes

• A tiny test set where a single case moves the score by several points.
• Only easy, document-derived questions, and no unanswerable ones.
• Scoring the final answer alone, so retrieval and generation faults cannot be told apart.
• Treating a model judge as ground truth without calibrating it.
• Re-running until the numbers look good, or editing the test set after a failure.

Where to go next

If you are choosing a starting point for a RAG project, the RAG boilerplates category lists the options, and this comparison explains how two of them differ. Whatever you choose, build your own test set first and judge any starter by how it performs on it.