RAG explained: how retrieval works and when to use it

Retrieval augmented generation, or RAG, makes a language model answer from your own documents. Before the model writes, a search step finds the few passages most related to the question and puts them into the prompt. That keeps answers current without retraining and lets every answer point to its source. It is not always needed: if your material fits comfortably in the prompt, you can skip it.

The name comes from a 2020 paper by Patrick Lewis and eleven coauthors, Retrieval Augmented Generation for Knowledge Intensive NLP Tasks, accepted at NeurIPS 2020. Its starting point still holds: models store facts in their parameters, but they struggle to access them precisely, to show where an answer came from, or to update what they know. The paper's fix was to pair the model, which it calls parametric memory, with a searchable index of documents, its non parametric memory.

The details are concrete. The authors split a December 2018 Wikipedia dump into 100 word chunks, about 21 million documents, turned each into a vector, and searched them with a FAISS index (the full paper). They also showed what they called index hot swapping: replace the index and the model's knowledge changes, with no retraining. That property is most of why businesses use RAG today.

One clarification before going further. The 2020 system trained its retriever and generator together. What most teams now call RAG is looser: an off the shelf embedding model, a vector database and a prompt that says "answer from these passages". The rest of this post is about that everyday version.

The indexing phase happens ahead of time. Anthropic's contextual retrieval post lays it out in three steps: break the documents into chunks of usually no more than a few hundred tokens, turn each chunk into an embedding, and store the embeddings in a vector database that can search by meaning. At question time the database finds the chunks closest to the question, and those chunks go into the prompt.

An embedding is the part that does the matching. OpenAI's embeddings docs define it as a list of floating point numbers where the distance between two lists measures how related the texts are: small distance, closely related. Its text-embedding-3-small model returns 1,536 numbers per text by default and text-embedding-3-large returns 3,072, each taking up to 8,192 tokens of input. The embedding model is itself a trained network, which is where how neural networks learn comes in.

The vector database is the part you choose. If you already run Postgres, pgvector adds vector search to it as an extension, with exact and approximate nearest neighbor search and cosine, inner product and L2 distance among its options. Its README is honest about the trade: approximate indexes trade some recall for speed, and the HNSW index searches better than IVFFlat at the cost of slower builds and more memory.

In practice the whole store can be one table. A sketch built from the commands in the pgvector README:

CREATE EXTENSION vector;

CREATE TABLE chunks (
  id        bigserial PRIMARY KEY,
  source    text,             -- which document, for the citation
  content   text,             -- the chunk itself
  embedding vector(1536)      -- text-embedding-3-small's default size
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);

-- at question time: embed the question, then fetch the closest chunks
SELECT source, content
FROM chunks
ORDER BY embedding <=> $1     -- cosine distance to the question's vector
LIMIT 20;

If the right passage never reaches the prompt, the model cannot use it, however good the model is. Anthropic gives a clean example of why: someone searches a support database for "Error code TS-999". An embedding model finds text about error codes in general and can miss the one chunk with that exact string, which an older keyword method called BM25 finds easily (Anthropic). The RAG paper saw the same thing: on fact checking claims that hinge on named entities, a plain BM25 retriever did best (Lewis et al.).

Anthropic measured how much fixing retrieval is worth. Adding a short, chunk specific explanation to every chunk before indexing, and running embeddings and BM25 together, cut the top 20 retrieval failure rate by 49%; adding a reranking step on top cut it by 67% (contextual retrieval). In the same tests, combining embeddings with BM25 beat embeddings alone, and passing the top 20 chunks to the model worked better than the top 10 or top 5.

Chunking is the quiet cause of many misses. A chunk cut from the middle of a contract may not say which contract or which clause it belongs to, which is exactly the gap the contextual step fills. Keeping headings, document titles and dates with each chunk costs little and gives both the search and the model something to hold on to.

Sometimes the best retrieval is none. Anthropic's advice is that a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt with no RAG, and that prompt caching makes repeating it cheaper and faster (Anthropic). Context windows have grown a lot since: current Claude models take 1M tokens per request (Anthropic models overview), and how large language models differ compares what other vendors offer.

A bigger window does not make retrieval obsolete, though. Anthropic's context window docs warn that accuracy and recall degrade as the token count grows. Google's long context guide puts numbers on the trade: you can get about 99% on a single lookup, but you pay the full input cost every time you send the query, so retrieving 100 separate facts that way means about 100 requests, and it adds that accuracy drops when the model must find several facts at once.

Put together, the choice usually follows the shape of the material:

• A small, stable set of documents that fits with room to spare: put it in the prompt and cache it.
• A collection larger than the window, or one that changes daily: use RAG, since updating means reindexing, not retraining.
• Answers that must cite the exact passage: RAG, because every retrieved chunk carries its source.
• Thousands of questions a day over the same corpus: RAG usually costs less, since each question sends a few chunks instead of everything.
• A task that needs the whole document at once, such as comparing every clause of one contract: long context, because chunks cannot show the whole.

Token cost is the thread running through all five. The same arithmetic behind cutting Claude Code token usage applies here: whatever you put in the prompt, you pay for on every call.

A first version does not need a framework. These steps, in this order, cover most internal knowledge bases and support assistants:

1. Collect the documents people actually ask about, and drop outdated versions before indexing.
2. Chunk them to a few hundred tokens, keeping the document title and section heading with each chunk.
3. Embed every chunk with one embedding model and store the vectors with their source in your database.
4. Search with embeddings and keyword matching together, take the top 20, and rerank if quality needs it.
5. Tell the model to answer only from the supplied passages, cite them, and say so when they do not contain the answer.
6. Test with real questions and score retrieval on its own, before judging the answers.

RAG also slots straight into agents. Anthropic counts retrieval as one of the three augmentations, with tools and memory, that turn a plain model into the building block of agentic systems, so a search tool over your documents is often the first tool an agent gets. Our agentic AI explainer shows where that fits in the loop.

More explainers like this live in the AI explained series, and production builds are in the AI agents category. If you want RAG over your own documents, wired into the tools your team already uses, that is part of our custom AI agents service.