Designing an Evidence-Based Answering System

Why this matters

A service-desk assistant gives a clear answer about renewing a VPN certificate: start the process 30 days before expiry. The answer sounds useful, but the current procedure says 14 days. The assistant has combined a familiar task with an obsolete instruction. Rephrasing the question does not fix the underlying problem: the answer needs evidence from a document that the model may never have seen.

Retrieval-augmented generation, or RAG, introduces a way to supply that evidence at the time of the question. The important engineering work is deciding which information is eligible, finding the right passage, and checking whether it actually supports the answer. A fluent paragraph is the final presentation, not proof that those earlier steps succeeded.

Throughout this course, you will use authored training documents for a hypothetical service desk. They are learning fixtures, not real operating policies. The numbers and failure cases are deliberately inspectable, so you can explain each result instead of relying on a demonstration that only works once.

The idea

Think of RAG as an open-book answering process. A question arrives. A retrieval component searches a collection of documents. The system selects a small evidence set and includes it in the model's input. The model then proposes an answer using that context. RAG describes the connection between retrieval and generation; it does not require one particular database, framework, embedding model, or vendor.

There are two separate paths. The preparation path turns source documents into searchable records. It extracts text, records where that text came from, splits long material into passages, and builds an index. The answer path handles a specific user's question: determine permitted sources, retrieve candidates, select context, request an answer, and validate the result. Keeping these paths separate helps you update documents without redesigning the question flow.

The retriever and generator have different responsibilities. A retriever ranks possible evidence. Its score measures something about the search method; it does not measure whether the final answer is true. A generator turns supplied information into language. It can still ignore evidence, combine incompatible versions, or add unsupported details. Even a well-designed pipeline must allow an abstention: a response explaining that the available sources do not answer the question.

RAG also differs from fine-tuning. Updating an index changes which external passages are available. Fine-tuning changes a model's parameters and is often used to influence behavior or task performance. These approaches can be combined, but fine-tuning is not a substitute for maintaining a source register with versions and access rules. If the issue is one revised procedure, changing the source collection is a more direct experiment than expecting model weights to become a document-management system.

For the practical work, use Python 3.10 or later and download the complete RAG lab. It uses the standard library and makes no network requests. You will build and inspect the retrieval and evidence-pack stages. It deliberately does not pretend that a language model generated an answer. A real model can be connected later, after you understand what information it would receive and how its responses would be evaluated.

Diagram

Worked example

The training collection contains six chunks. Four are current staff documents: VPN renewal, password reset, backup retention, and incident escalation. One is an older VPN document. One is a restricted recovery document available only to the security group. That collection is tiny, but it contains three important complications found in larger systems: an outdated answer, a related but insufficient answer, and information the requester may not access.

The requester belongs to the staff group and asks, “How do I renew my VPN certificate?” Start by defining an acceptable answer. It should identify the device portal, the Renew certificate action, the 14-day window, and the instruction to restart the VPN client. It must cite the current VPN passage. It should not include recovery-console instructions from the restricted document. Those are concrete acceptance conditions that a reviewer can check.

Now follow the pipeline. The access and active-version filter reduces six chunks to four. The keyword retriever scores the eligible chunks and ranks the current VPN passage first. The password-reset passage may also match because it mentions VPN certificates, but its purpose is to distinguish password reset from certificate renewal. Retrieving both is not automatically a mistake; treating them as equally sufficient answers would be.

Suppose you reserve a context allowance of 100 words. The lab packs whole passages in ranking order, without repeating the same source. It never cuts a passage in the middle merely to fill the allowance. A real integration must use the selected model's tokenizer and also reserve room for instructions, the question, and the answer. Here, a word budget makes the exercise transparent and reproducible without downloading a tokenizer.

The output is a structured evidence pack containing an instruction, the original question, passage IDs, titles, and text. Its status is EVIDENCE_RETRIEVED. That label means passages were selected. It does not mean the question was answered correctly. This distinction prevents a search match from being presented as a completed evaluation.

Try a second question: “What is the lunch menu?” None of the fixture documents contains that information. The lab returns NO_EVIDENCE. The correct next step is to explain the missing evidence or direct the requester to an appropriate source. Generating a plausible menu would produce a more complete-looking answer while making the system less useful.

Finally, consider the 30-day error that opened this lesson. If the old document is excluded before retrieval, the generator cannot receive that passage through this pipeline. If the old document is still included, a stronger model might notice the conflict, but your source-management problem remains. Fixing the stage that introduced the error is easier to test than adding another sentence to the prompt and hoping it overrides the problem.

Code

Save the lab in a working folder. In a terminal opened in that folder, run:

python3 rag_lab.py --stage ingest
python3 rag_lab.py --stage pack
python3 rag_lab.py --stage test

The first command lists six fixture records and their access metadata. The second prints the evidence pack. The final command runs the bundled checks. On Windows, use py -3 instead of python3 if that is how Python is installed. These commands inspect real local computation; they do not call an AI service.

Common mistakes

Treating retrieved text as a guaranteed answer. This is tempting because the highest-ranked passage looks more relevant than the others. But ranking is comparative: first place can still be a weak match. Read the passage against the precise question and separate “candidate evidence found” from “claim supported.” Otherwise a document that merely repeats the question's vocabulary can become an authoritative-sounding response.

Assuming more context always improves results. Adding every possible passage feels safer than leaving information out. In practice, old versions, repeated text, and loosely related material compete with the evidence that matters. Start with a small, inspectable selection and measure whether additional context improves answers. A larger context window is a capacity limit, not a promise that every passage will be used well.

Measuring only the final paragraph. A correct answer in a demo can hide poor retrieval if the model already knows the subject. Use questions whose answers come from the supplied documents, inspect the retrieved IDs, and include unsupported questions. If you cannot identify which source supports a claim, you cannot tell whether the system used the intended evidence or produced a lucky answer.

Confusing the lab with a complete model service. A local evidence pack is a useful engineering artifact, but it does not test generation quality, streaming, provider limits, or model behavior. Keeping that boundary explicit makes your next experiment clearer: connect a model, preserve the evidence pack, and evaluate the generated answer separately. Do not report a retrieval test as an end-to-end language-model result.

Your turn

Write an acceptance checklist for an assistant that answers questions about a small collection of technical procedures. Include one supported question, one question that has no answer in the collection, and one question that asks for restricted information. For each, state which document IDs are permitted and what the system should do when evidence is insufficient.

Run the three commands above and inspect the evidence pack. Find the 14-day instruction and confirm that neither the archived 30-day rule nor the restricted recovery console appears. Then change the question in a Python session by calling evidence_pack('lunch menu', {'staff'}).

You succeed when you can explain the result of both queries without referring to the model's intelligence. Your explanation should identify the source-selection rule, the relevant or missing passage, and the expected next action. Keep this checklist: the final module turns it into a release test rather than leaving it as a design note.

Recap

• RAG supplies external evidence during answering; it does not make a language model inherently correct.
• Source preparation and per-question retrieval are separate processes with different failure modes.
• A retrieval score ranks candidates and must not be presented as answer confidence.
• An unsupported question needs an explicit missing-evidence response rather than a fabricated answer.
• This course's local lab computes retrieval and evidence packs; generation requires a separate integration and evaluation.

Sources

• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. The original research combines a retriever with a generator; this course uses a simpler application pipeline for inspection.
• Liu et al. (2023), Lost in the Middle: How Language Models Use Long Contexts. Experiments showing that relevant information can be harder to use at some positions in long contexts; not a guarantee about every current model.