Building a Simple RAG Pipeline Step by Step

You understand chunks, embeddings, and search. Now let's wire them together into one pipeline — like connecting LEGO blocks into a working toy robot that answers from your notes.

What Is a RAG Pipeline?

A RAG pipeline is the end-to-end path from raw documents to grounded answers. It has an offline indexing path (prepare knowledge) and an online query path (answer users).

Offline: Build the Knowledge Index

Documents
   ↓ extract text
Chunks + metadata
   ↓ embedding model
Vectors stored in search index

Online: Answer a Question

User question
   ↓ embed question
Retrieve top-k chunks
   ↓ build prompt with context
LLM generates answer + optional citations

Step-by-Step: Minimal C# Flow

Step 1: Index documents (run once or on schedule).

Step 2: On each question, search the index:

var chunks = await searchClient.SearchAsync(userQuestion, top: 5);
var context = string.Join("\n---\n", chunks.Select(c => c.Text));

Step 3: Call chat with strict system prompt:

var messages = new[]
{
    new ChatMessage(ChatMessageRole.System,
        "Answer only from CONTEXT. If unsure, say 'I don't know.'"),
    new ChatMessage(ChatMessageRole.User, $"CONTEXT:\n{context}\n\nQUESTION: {userQuestion}")
};
var answer = await chatClient.CompleteChatAsync(messages);

Step 4: Return answer plus source links from chunk metadata.

Real-World Example

A campus FAQ bot indexes registrar PDFs nightly. Students ask about exam forms at midnight. Retrieval finds the 2026 dates chunk; the model summarizes with a link — no administrator awake required.

Common Misconceptions

"Indexing once is enough." Stale indexes lie politely. Automate refresh.

"More chunks always help." Too much context confuses models and costs tokens.

Offline vs Online Jobs

Offline indexing runs on a schedule or when files change — heavy embedding work happens here.

Online query path must stay fast — embed question, search, call LLM, return under a few seconds for chat UX.

Never re-index entire corpora on every user click; users will abandon the bot before the spinner stops.

Reusable Prompt Template

var prompt = "CONTEXT:\n" + context +
    "\n\nQUESTION:\n" + question +
    "\n\nAnswer using CONTEXT only. Cite sources. If unknown, say so.";

Latency Budget

Target under three seconds total: 200 ms embed query, 300 ms search, 2 seconds LLM generation. Log each segment. If search is slow, index is too big or unoptimized; if LLM is slow, reduce context length or use a faster model.

Version your prompts in Git like application code. When answer quality shifts, diff prompt changes alongside model version bumps. Mystery regressions often trace to someone editing a system message in the portal without telling the team — version control prevents silent drift.

Caching Retrieved Context

Identical repeated questions within minutes — 'What are office hours?' — can cache retrieval results briefly to save embedding and search cost. Invalidate cache when indexer runs. Do not cache personalized answers tied to user-specific documents without including user id in cache key.

Separate admin reindex endpoint from public chat endpoint. Accidentally exposing reindex to anonymous users invites denial-of-wallet attacks burning embedding credits. Authentication and rate limiting belong in RAG APIs same as any production REST service.

Summary

A simple RAG pipeline is two workflows: build the index, then retrieve-and-generate per question. Master this skeleton before adding reranking, agents, or fancy UI.