RAG Evaluation Guide: How to Test Retrieval-Augmented Generation Apps

Overview

RAG evaluation is the process of testing both halves of a retrieval-augmented system: the retrieval layer that selects source material and the generation layer that uses that material to produce an answer. Testing only the final response can hide the cause of a failure. A wrong answer may result from missing source content, poor chunking, weak ranking, an unclear prompt, or an LLM that ignored relevant context.

A useful evaluation plan separates these questions:

• Retrieval quality: Did the system return the passages needed to answer the question?
• Answer accuracy: Is the response correct according to the reference answer or accepted source?
• Groundedness: Are the response's claims supported by the retrieved context?
• Citation coverage: Are important factual claims linked to appropriate sources?
• Operational performance: Is latency predictable, and is the cost acceptable for the intended workload?

These measures should be viewed together. A fluent answer with weak evidence is not a successful RAG result. Likewise, excellent retrieval does not help users if the final response is confusing, incomplete, or too slow.

Start with a small, representative test set rather than an abstract benchmark. Include straightforward questions, multi-part questions, questions requiring details from more than one document, ambiguous wording, unanswerable questions, and questions about recently changed material. For content teams and publishers, the test set should reflect the actual questions readers ask about the knowledge base, archive, product documentation, or editorial workflow.

How to estimate

Use a spreadsheet or evaluation script with one row per test case. A practical row can include:

case_id | question | expected_answer | required_sources | retrieved_chunks | generated_answer | citations | retrieval_score | groundedness_score | latency | estimated_cost | failure_type

For each question, record whether the required evidence appeared in the retrieved results. If the system returns ranked passages, note the position of the first relevant passage. This makes it possible to compare retrieval settings such as chunk size, overlap, metadata filters, reranking, and top-k selection.

For answer review, use a consistent rubric rather than an overall impression. A simple three-point scale works well:

• 0 — Fail: The answer is incorrect, unsupported, or does not address the question.
• 1 — Partial: The answer is useful but misses a material detail, includes an unsupported claim, or cites only part of the evidence.
• 2 — Pass: The answer is accurate, appropriately limited to the evidence, and sufficiently complete.

Calculate a score for each dimension separately. For example, answer pass rate can be calculated as:

answer pass rate = passed answer cases / total answer cases

Citation coverage is more precise when measured at the claim level:

citation coverage = supported factual claims with valid citations / total factual claims requiring citations

For cost estimation, use the inputs available in your own deployment rather than relying on a fixed example. A basic monthly estimate is:

monthly cost = request volume × average cost per request + fixed infrastructure cost

Break average request cost into embedding, retrieval, reranking, model generation, storage, and monitoring components where applicable. If pricing changes, update the input cells rather than rewriting the calculation. For a broader evaluation framework, compare this process with the scorecard approach described in How to Evaluate LLM Output Quality with a Repeatable Scorecard.

Inputs and assumptions

Your results are only as useful as the assumptions behind them. Document the following before comparing two RAG configurations.

Test-set composition

Record the number of cases and the distribution of question types. A set made entirely of easy factual lookups can make a weak system appear reliable. Mark cases that are ambiguous, unanswerable, time-sensitive, or dependent on multiple sources. Keep a private holdout set for final checks so that prompt or retrieval tuning does not optimize only for questions the team has already inspected.

Reference standard

Define what counts as a correct answer. The reference may be a source passage, a structured answer key, or a set of required facts. For open-ended questions, list acceptable points rather than demanding identical wording. Also define how the system should respond when the evidence is missing. In many applications, a clear statement that the available sources do not establish an answer is preferable to an unsupported completion.

Retrieval settings

Capture the index version, document version, chunking method, embedding configuration, filters, top-k value, and reranking settings. Changing any of these can alter results. If documents contain tables, images, or scanned pages, record whether extraction and OCR were applied. A RAG system cannot retrieve information that was never represented correctly in the index; document processing may therefore deserve its own test cases. See Best OCR APIs for Documents, Screenshots, and Images for related workflow considerations.

Generation settings

Record the model, system instructions, prompt version, context limits, temperature or equivalent controls, output format, and citation rules. Treat prompts as versioned components. A small instruction change can affect refusal behavior, citation completeness, verbosity, and the tendency to rely on prior model knowledge. The prompt engineering workflow at Prompt Engineering Workflow: A Reusable Framework for Reliable AI Outputs can help establish that discipline.

Operational assumptions

Measure latency using a clearly defined boundary, such as time from request receipt to complete response. Separate retrieval latency from generation latency when possible. Record errors, timeouts, retries, token usage, and concurrent load. Cost and latency averages can hide poor experiences, so include a percentile or worst-case view when your tooling supports it.

Worked examples

Assume an illustrative test set contains 40 questions. Twenty-eight require a single source, eight require information from multiple passages, and four are intentionally unanswerable. These numbers are examples only; replace them with the shape of your own workload.

During the first run, relevant evidence appears in the retrieved context for 34 cases. The retrieval evidence rate is therefore:

34 / 40 = 85%

The final answers pass review for 30 cases, giving an answer pass rate of 75%. Review shows that five failures occurred even though the right evidence was retrieved. Those are generation or prompt failures, not retrieval failures. The remaining five failures involve missing or poorly ranked evidence and should be investigated in the indexing or retrieval layer.

Next, reviewers identify 52 factual claims across the 30 evaluated answers. Forty-six claims have citations that support them:

46 / 52 = 88.5% citation coverage

This result does not mean the system is 88.5% accurate. Citation coverage measures whether claims are supported, while answer accuracy measures whether the response addresses the question correctly. Keep the measures separate to avoid masking a serious weakness.

Suppose an alternative configuration improves evidence retrieval from 85% to 90% but increases average latency and cost. Do not adopt it based on one number. Compare the change by question type, inspect newly introduced failures, and estimate its effect at expected request volume. If the system handles 10,000 requests per month, a small per-request increase should be multiplied by that volume before a decision is made. Recalculate with the current internal pricing inputs whenever provider rates, model selection, request length, or traffic assumptions change.

Use a failure taxonomy to turn observations into action:

• Ingestion failure: Content was missing, stale, truncated, or extracted incorrectly.
• Segmentation failure: A relevant fact was split from the context needed to interpret it.
• Retrieval failure: The correct passage was not returned or was ranked too low.
• Context failure: Relevant material was retrieved but diluted, truncated, or conflicting.
• Generation failure: The model misread, ignored, or contradicted the supplied evidence.
• Presentation failure: The answer was substantially correct but unclear, incomplete, or poorly cited.

When to recalculate

Run the evaluation again after any change that could alter evidence, instructions, model behavior, or operating economics. At minimum, trigger a regression run when you add or remove source documents, rebuild the index, change chunking or embeddings, modify metadata filters, alter top-k or reranking, update the system prompt, switch models, change output or citation formatting, or add a new user workflow.

Recalculate cost when model or infrastructure pricing inputs change, but also when request volume, average context length, output length, retry rates, or concurrency changes. A configuration that was affordable during a pilot may behave differently at production volume. Keep a dated record of assumptions so that later comparisons are meaningful.

Make monitoring continuous for live applications. Sample real requests, remove sensitive information where necessary, and review failures using the same taxonomy as the pre-release test. Track unanswered questions, citation gaps, user corrections, latency spikes, and changes in document freshness. A small, stable regression set can run on every release, while a broader review can run on a regular schedule or after a major content refresh.

For a practical release checklist, confirm that the test-set version, source snapshot, prompt version, model, retrieval settings, scores, latency, and estimated cost are all recorded. Then compare the new run with the previous baseline, investigate regressions by failure type, and approve the change only when the trade-offs are understood. This makes RAG evaluation an ongoing quality-control process rather than a one-time demonstration.