deskthebuildRAG
← All notes · 5 min read

Our RAG system retrieves relevant documents but still answers incorrectly.

This is a hypothetical production scenario: search results look relevant, yet the generated answer is wrong. “Relevant” is not sufficient evidence that retrieval augmented generation is working. The required fact may be absent, buried, stale, contradictory, or presented to the model without enough structure to use it correctly.

Why the common answer is incomplete

Most candidates answer: “Increase top-k or use a better embedding model.”

That may improve recall, but it can also add noise and contradictions. A document can be semantically related to the question without containing the exact evidence needed to answer it. Even perfect retrieval cannot compensate for incorrect source data, bad chunk boundaries, context truncation, or generation instructions that encourage the model to answer beyond the evidence.

Debug RAG as a pipeline with separate retrieval and answer-quality contracts.

A production approach

1. Define the evidence required for the answer

For each failing question, identify the authoritative source, exact passage or fields, version, and expected answer constraints. Label whether the system should answer, ask for clarification, or abstain. This turns “the documents seem relevant” into a concrete test: did the candidate set contain the required evidence, and did the final context preserve it?

2. Inspect data before search

Verify that ingestion produced the correct text, metadata, permissions, timestamps, and document version. Parsing can drop tables, merge columns, omit headings, or duplicate boilerplate. Access-control filters can remove the authoritative document. If the indexed representation is wrong, changing embeddings only makes the wrong representation easier to retrieve.

3. Measure retrieval in stages

Evaluate candidate recall before reranking, recall after filters, ranking position after reranking, and inclusion in the final context. Record document and chunk identifiers at every stage. A relevant document at rank two does not help if a token budget truncates it, and a required chunk can disappear when metadata filters use the wrong tenant or date.

4. Fix chunking around meaning

Chunk boundaries should preserve the units needed to answer questions: a policy with its exceptions, a function with its contract, or a table with its header. Use document structure when available and include parent titles or section paths. Overlap can preserve boundary context but increases duplication and index size. Test chunking using real questions rather than selecting a universal character count.

5. Align query construction with the corpus

User language may not match the source vocabulary. Expand acronyms, resolve entities, carry necessary conversation references, and separate compound questions when appropriate. Hybrid lexical and semantic retrieval can help exact identifiers and paraphrased concepts. Query rewriting should remain observable; an incorrect rewrite can confidently search for the wrong thing.

6. Rerank for answer evidence

Rerank a bounded candidate set using the question and passage, not document popularity alone. Train or select the reranker for the corpus and language when data supports it. Preserve diversity when sources may conflict. More candidates improve recall but increase reranking latency and cost, so choose the cutoff through evaluation rather than habit.

7. Package context explicitly

Deduplicate passages, retain source titles and dates, separate documents with clear boundaries, and place the strongest evidence where the model can use it. State which sources are authoritative when policies conflict. Do not concatenate tool errors or retrieved instructions into the system-instruction region. Treat retrieved text as untrusted data and keep it distinct from application policy.

8. Constrain and evaluate generation

Require the answer to cite evidence identifiers that the application can verify. Instruct the model to state when the supplied context is insufficient or conflicting. Evaluate factual support sentence by sentence, not only semantic similarity to a reference answer. Run retrieval-only and generation-with-gold-context tests separately: gold-context failure implicates generation, while retrieval failure implicates earlier stages.

Engineering trade-offs

Larger candidate sets increase recall while raising latency, cost, and noise. Smaller chunks improve match precision but can lose relationships; larger chunks preserve context but dilute the signal. Hybrid retrieval improves exact-match behavior but adds indexing and tuning complexity. Strict grounding reduces unsupported answers but increases abstention. Fresh indexing improves correctness while consuming ingestion capacity and complicating consistency.

The right configuration depends on corpus structure, question types, freshness requirements, and the cost of a wrong answer.

Common mistakes

  • Calling a whole document relevant without locating the required evidence.
  • Evaluating only final answers and never measuring retrieval recall.
  • Increasing top-k until the prompt is full.
  • Chunking tables without their headers or policies without exceptions.
  • Mixing content from different tenants in retrieval or cache keys.
  • Treating retrieved text as trusted instructions.
  • Changing embeddings, chunking, and prompts simultaneously so the result cannot be attributed.

Failure handling and observability

Trace ingestion version, query and rewrite, applied filters, candidate IDs, retrieval scores, reranking scores, final context IDs, token truncation, citations, and answer-validation results. Track candidate recall, final-context recall, citation validity, groundedness, abstention, stale-source use, latency, and cost. Slice metrics by corpus, document type, language, question type, and version.

If retrieval returns no reliable evidence, do not silently fall back to model memory for a domain that requires grounding. Ask for clarification, return a bounded “not found,” or route to an authoritative tool. If sources conflict, expose the conflict with dates and identifiers. Preserve failed traces long enough for debugging under an explicit privacy and retention policy.

Interview insight

The interviewer is not testing whether you have used a particular vector database. They are testing whether you can separate ingestion, retrieval, reranking, context assembly, and generation. A strong answer finds the first stage that lost the required evidence and improves that stage with a measurable evaluation.