RAG Quality Starts with Retrieval Evidence

Sep 5

By

ai systemsragevaluationsearch

When a retrieval-augmented generation system gives a wrong answer, “the model hallucinated” is often an incomplete diagnosis. The correct evidence may be missing from the corpus, missed by retrieval, removed during context assembly, ignored by generation, or cited incorrectly.

RAG is a pipeline. Quality has to be attributed to a stage.

source -> parse -> chunk -> index -> retrieve -> rerank
       -> assemble context -> generate -> verify citation

Define a traceable evaluation unit

For each test case, store:

  • user question and relevant cohort;
  • expected answer or decision criteria;
  • authoritative source document and passage;
  • corpus/index version;
  • retrieved document and chunk IDs with scores;
  • final context after truncation;
  • model, prompt, and generation parameters;
  • answer and citation verdict.

Without this lineage, two runs that look comparable may have used different evidence.

Evaluate stages independently

Corpus coverage

Can the authoritative source be found in the indexed corpus? Parsing failures, access-control filters, stale synchronization, and poor chunk boundaries are corpus problems. A better language model cannot recover evidence it never receives.

Retrieval

Measure whether relevant passages appear in top-k candidates. Use recall@k and rank-aware metrics, then slice them by query class, language, document age, and filter combination. A global average can hide a complete failure for one tenant or content type.

Context assembly

Record which retrieved passages survive reranking, deduplication, token limits, and policy filtering. “Retrieved” does not mean “present in the prompt.”

Generation

Given a context that definitely contains the answer, does the model answer faithfully, express uncertainty, and refuse unsupported conclusions? This isolates generation from retrieval.

Citation

Check that each material claim is entailed by the cited passage and that links resolve to a source the user may access. Citation formatting alone is not groundedness.

Use a failure taxonomy 

NO_SOURCE        authoritative material absent
STALE_SOURCE     indexed version too old
RETRIEVAL_MISS   relevant passage outside candidates
RERANK_DROP      candidate removed before context
CONTEXT_TRUNCATE relevant text exceeded budget
GENERATION_DRIFT answer contradicts supplied evidence
CITATION_MISS    claim lacks supporting passage
ACL_LEAK         retrieved content violates access policy

Assign one primary failure and optional contributing failures. Weekly counts then tell the team whether to invest in ingestion, search, prompts, models, or authorization.

Production gates

Before release, require thresholds for high-risk cohorts, not only aggregate scores. Validate source freshness, tenant isolation, latency, cost, and abstention behavior. In production, sample traces for human review and maintain canary questions whose answers should change when source content changes.

Common mistakes

  • Using answer similarity as the only metric.
  • Evaluating with synthetic questions that mirror chunk wording.
  • Logging prompts without corpus and index versions.
  • Letting the model cite a document ID it never received.
  • Increasing context size before measuring retrieval misses.
  • Mixing permission filtering after retrieval with a shared candidate cache.

Trade-offs

Larger k can improve recall while increasing reranking cost and irrelevant context. Smaller chunks improve passage precision but lose surrounding meaning and expand index size. Human labels are expensive but expose failures that model-based graders can reproduce or amplify. More detailed traces improve diagnosis while increasing privacy and retention obligations.

The useful question is not “Is our RAG accurate?” It is “Which stage failed, for which cohort, under which version, and what evidence proves it?”

Further reading

>