RAG Engineering 01: How RAG Works, from Documents to Answers

Retrieval-augmented generation (RAG) connects an LLM to information outside its training data. The application searches documents or records, puts relevant passages into the prompt, and asks the model to answer using that context.

For an assistant that answers questions about internal APIs, product manuals, or company policies, the engineering work is deciding what to retrieve, what to send to the model, and how to connect the answer back to its sources.

RAG: from sources to cited answers

Scroll to follow the diagram, or open the interactive view →

Explore interactive diagram · ArchifyEditable Archify specification

What retrieval adds to a prompt

A language model can answer from its learned parameters and the context you provide. It cannot automatically read a private repository or discover which version of an internal policy is current. Your application needs to supply that information.

Retrieval automates the selection you would otherwise do by finding a document and pasting it into the chat. That becomes useful when the document collection is too large or changes too often for manual selection.

The original RAG paper studied a trainable retriever–generator combination. This series uses the broader application pattern: retrieve external information and make it available during generation. Joint training is not required.

Two paths: build the index, then answer questions

Document preparation runs when sources are added or changed. Query processing runs for each question. Keeping these paths separate lets you update documents without repeating parsing and embedding work for every request.

The published index joins these paths. Explore the full architecture diagram to see the data flow and authorization boundary.

Follow a passage into an answer

Use a small policy passage to see exactly what crosses each boundary. The following is an illustrative example:

The application assigns C1 to a known source passage. When the answer refers to it, the application resolves that ID to the original document version and section. A generated citation is useful only when that connection exists.

A minimal evidence record keeps the identities separate:

{
  "chunk_id": "support-v2:p1:0",
  "document_id": "atlas-support",
  "version_id": "support-v2",
  "locator": "Priority-one incidents",
  "text": "Atlas acknowledges priority-one incidents within 15 minutes."
}

The document ID identifies the source, the version ID identifies its revision, and the chunk ID identifies the retrieved unit. The locator gives the reader somewhere to inspect the claim.

Follow the implementation on GitHub

My implementation has two entry paths: a small in-memory pipeline for fixture documents, and a collection-backed service with persisted ingestion, indexing, and retrieval. Start with the smaller path so each transition is visible.

Excerpt from pipeline.py. This is part of the application, not a standalone script.

self.prepare()
assert self._vector_store is not None
started = perf_counter()
matches = self._vector_store.similarity_search_with_score(
    question, k=top_k or self.default_top_k
)
elapsed = _elapsed_ms(started, perf_counter())
retrieved = [
    RetrievedChunk(
        chunk=langchain_document_to_chunk(document),
        rank=rank,
        score=float(score),
    )
    for rank, (document, score) in enumerate(matches, start=1)
]
return retrieved, elapsed

prepare() loads documents, splits them, and builds an InMemoryVectorStore once for this pipeline instance. Each query returns the chunk, its rank, its similarity score, and retrieval latency. Those are useful debugging artifacts before inspecting any generated answer.

Read index preparation · Read the generation handoff · Read evidence and citation construction

The handoff constructs request-local citation IDs and carries the original chunk text into GenerationService. In this baseline, sufficient=bool(retrieved) only checks that retrieval returned something. The collection-backed retrieval service adds coverage heuristics, context packing, and a bounded corrective attempt; Part 6 examines those decisions.

Retrieve broadly, send selectively

Finding ten candidates does not mean sending ten passages to the model. Some may overlap, repeat the same fact, or consume the budget without helping answer the question.

In a larger system, apply the caller’s access scope before selecting candidates, then pack distinct source regions within the context allowance. The in-memory pipeline above isolates the basic retrieval-to-generation path; the later articles follow the collection-backed implementation.

Two checks matter independently: did retrieval find the required passage, and did context assembly keep it? If the evidence disappeared before generation, a different answer prompt will not recover it.

Where an answer can go wrong

Use the source-to-answer path to locate the first failing boundary. Barnett et al. describe failures across this pipeline; the cards below turn that idea into practical debugging questions.

Choose the intervention that matches the problem

These approaches can be combined. Fine-tuning can improve evidence use, but it does not remove the need to update source facts. Retrieval can supply a current policy, but the model can still misinterpret it.

For a first implementation, use a small corpus whose answers you can inspect. Record the source version, chunker and embedding settings, retrieved IDs, final context, and prompt. Those records make a change explainable when you later add hybrid search, reranking, or a larger model.

The next article starts with the earliest representation decision: where to split a document so a relevant fact keeps the context needed to interpret it.