What Is RAG? Retrieval-Augmented Generation Explained

If you have ever asked an AI tool a question about your own company's documents and gotten a confident, wrong answer, you have run into the exact problem RAG was built to fix. This post is for founders and product leads who keep hearing "RAG" in vendor pitches and want to know, concretely, what it is, how it works, and when it actually earns its complexity.

What is RAG (retrieval-augmented generation)?

RAG is a way of answering questions by combining two systems: a search step that finds relevant passages from your own data, and a generation step where an LLM writes the answer using those passages as context. Instead of asking the model to answer purely from what it learned during training, you hand it the specific facts it needs, every time, right before it answers.

The term comes from a 2020 Meta AI research paper, but the pattern it describes predates the name: search first, then let a language model synthesize the result into a readable answer. What changed since 2020 is that embeddings and vector search got cheap and fast enough to do this at production scale, and LLMs got good enough to write fluent, grounded answers from retrieved text instead of just summarizing search snippets.

How does a RAG pipeline actually work?

A RAG pipeline has five steps, and most production failures trace back to getting one of them wrong.

1. Ingest and chunk. Your documents (PDFs, wiki pages, support tickets, contracts) get split into smaller pieces, typically 200-800 tokens each. Chunk too large and retrieval gets imprecise; too small and you lose context the model needs to answer correctly.
2. Embed. Each chunk is converted into a vector, a list of numbers that represents its meaning, using an embedding model (OpenAI's text-embedding-3, Google's Gemini embedding models, or an open-source model like BGE or E5 that you can run yourself).
3. Store and index. Vectors go into a vector database such as pgvector, Pinecone, or Weaviate, which can find the nearest vectors to a query in milliseconds even across millions of chunks.
4. Retrieve. When a user asks a question, it gets embedded the same way, and the database returns the top-k most similar chunks (commonly 3-10). Stronger pipelines also re-rank these results with a smaller cross-encoder model before passing them on, and some add keyword search alongside vector search ("hybrid search") because pure semantic similarity misses exact terms like part numbers or legal clause references.
5. Generate. The retrieved chunks are inserted into the LLM's prompt along with the user's question and instructions to answer only from the provided context. The model writes the final answer, ideally with citations back to the source chunks.

Everything upstream of generation determines whether the LLM even has a chance to get the answer right. A brilliant model fed the wrong chunks will confidently produce a wrong answer.

When do you actually need RAG?

You need RAG when your use case requires answers grounded in information the model was not trained on and that changes faster than you can retrain or fine-tune for. Common cases we build for clients: document-analysis tools that read tender or contract documents and answer questions about them (our i2ao project does exactly this for French public-tender documents), internal knowledge assistants, customer support bots that need current product docs, and RFP response tools that pull from a company's approved answer library, similar to what we built for replymagik.

You probably do not need RAG if the task is pure reasoning or writing with no dependency on private or fast-changing facts, if your knowledge base is small enough to fit entirely in the model's context window every time (a few thousand words), or if you need the model to learn a consistent style, format, or domain vocabulary more than it needs facts, in which case fine-tuning is the better tool, sometimes alongside RAG rather than instead of it.

What breaks RAG pipelines in production?

RAG looks simple in a demo with ten clean PDFs and falls apart with ten thousand messy ones. The common failure points, in order of how often we see them:

• Bad chunking. Splitting mid-sentence or mid-table destroys meaning. A chunk that says "the fee is" with the actual number in the next chunk is useless on its own.
• Retrieval misses the right chunk. Semantic search finds conceptually similar text, not necessarily the correct answer. A question about "cancellation fees" might retrieve a chunk about "termination clauses" that uses different wording and never surfaces, while the literally-relevant paragraph sits lower in the ranking.
• Stale indexes. If a document changes and the vector store is not re-indexed, the system confidently cites an outdated version. This is a pipeline design problem, not a model problem.
• Context window stuffing. Throwing in 20 chunks "to be safe" dilutes the model's attention and increases the chance it blends information from the wrong source, or ignores instructions buried in a long prompt.
• No source attribution. Without citations back to specific chunks, you cannot tell a hallucination from a correct answer without manually checking, which defeats the point. See our related post on preventing AI hallucinations for how we test for this.
• Treating retrieval as a one-time build. Document sets, user phrasing, and edge cases evolve. Pipelines that are not evaluated and tuned over time degrade quietly, often without anyone noticing until a customer catches a wrong answer.

The fix for most of these is evaluation discipline: a test set of real questions with known correct answers, measured before and after any pipeline change. See how to evaluate LLM applications for how we set that up.

What does a RAG system look like in a real build?

Take a document-analysis tool that reads public-tender documents, similar in shape to our i2ao project. A tender package arrives as a bundle of PDFs: the RC (contract rules), CCAP (administrative clauses), and CCTP (technical specifications), often 50-300 pages combined and inconsistently formatted. The pipeline has to handle a few things a toy demo never faces:

• Multi-document context. A single question like "what is the penalty for late delivery" might need information from both the CCAP and the CCTP, so retrieval has to search across the whole bundle, not one file at a time, and the generation step has to be told which document each chunk came from so it can cite correctly.
• Tables and structured clauses. Procurement documents are full of numbered clauses and pricing tables. Naive text chunking that ignores structure will split a table mid-row, so ingestion needs table-aware parsing, not just a generic text splitter.
• Language and terminology. French-language legal and procurement terms need an embedding model that handles the language well, and retrieval needs to work whether the user asks in formal or informal phrasing.
• Verifiable output. Because the analysis feeds a paid product used by procurement teams making real bid decisions, every generated answer needs a visible citation back to the clause it came from, so a human can check it in seconds rather than re-reading the whole document.

None of this is exotic engineering, but it is the difference between a RAG demo that works on a sample PDF and a RAG product that holds up against a procurement officer who has read the document twice and knows exactly what it says.

How do you measure whether a RAG system is actually working?

Three metrics matter more than the rest, and all three are measurable before you ship:

1. Retrieval recall. Out of a test set of real questions with known correct source chunks, what percentage of the time does retrieval actually return the right chunk in its top results? If recall is below roughly 85-90% on your test set, no amount of prompt tuning on the generation side will fix the downstream answers, because the model is working from the wrong material.
2. Answer faithfulness. Does the generated answer only state what the retrieved context supports, or does it add details the context never mentioned? This is typically graded by having a second LLM call (or a human reviewer) compare the answer against the retrieved chunks and flag unsupported claims.
3. Abstention accuracy. When the retrieved context genuinely does not contain the answer, does the system say so, or does it guess? A system that never says "I don't know" is far more dangerous in production than one that occasionally does, because users start trusting every confident-sounding answer equally.

We track all three against a fixed evaluation set every time the pipeline changes, whether that's a new chunking strategy, a swapped embedding model, or an added re-ranking step, so we can tell whether a change actually helped or just felt like it should.

RAG vs. just using a longer context window

Modern frontier models now accept context windows large enough to hold entire document sets, which raises the question of whether retrieval is even necessary anymore. In practice it still is, for three reasons: cost (sending the same 500-page manual on every single query multiplies your API spend for no benefit), latency (processing a huge context on every call is slower than retrieving a handful of relevant chunks), and accuracy (models still attend unevenly across very long contexts, and a well-targeted retrieval step often outperforms "just paste everything in" even when the context window technically fits). Long context and RAG are complementary, not competing: long context lets you retrieve more generously and worry less about chunk boundaries, it does not remove the need to retrieve selectively in the first place.

How much does a RAG system cost to build?

Costs vary with document volume, data cleanliness, and how much retrieval tuning the use case demands. Market-wide, production RAG chatbots typically run $15,000-$60,000 depending on scope and data complexity, and document-heavy or multi-source RAG agents can run higher. We've written a fuller breakdown in LLM integration cost. Ongoing costs are mostly embedding generation (cheap, usually under a few dollars per million tokens), vector database hosting, and the per-query LLM calls, which is the largest recurring line item for most teams.

How we approach RAG at Venzpire

We treat retrieval as the part of the system that determines whether the whole thing works, which means we spend real time on chunking strategy, hybrid search, and evaluation before we tune prompts. Our $1,500 discovery sprint maps your actual document set, tests retrieval against real questions, and surfaces the failure modes above before you commit to a full build. From there, production RAG systems typically run from $15,000, depending on data volume and integration complexity. If you're scoping a document-analysis tool, internal knowledge assistant, or customer-facing RAG system, start with our LLM integration page or apply directly.