Retrieval-Augmented Generation, explained

By 2020, large pre-trained language models had shown they absorb a surprising amount of factual knowledge during training. The paper’s abstract names three things they still could not do well. They could not reliably access and manipulate that knowledge, so on knowledge-intensive tasks they lagged behind systems built for the task. They could not show provenance, meaning where an answer came from. And they could not have their world knowledge updated without being trained again.

Models with a differentiable link to an external memory existed, but, as the authors put it, they had “only been investigated for extractive downstream tasks”: systems that find and copy a span of text from a document. RAG asks whether the same idea works when the model has to write the answer.

The paper frames this as two kinds of memory. The parametric memory is the generator’s weights. The non-parametric memory is a dense vector index of Wikipedia that the model searches. The second can be read, checked and replaced; the first cannot. A model that uses both should be able to keep the fluency of a large language model and gain the traceability of a search engine.

Given an input x (a question), RAG retrieves text passages z and uses them as extra context when generating the output y. There are two components, both taken off the shelf and then fine-tuned together.

The retriever is Dense Passage Retrieval (DPR), a bi-encoder: one BERT-base model encodes documents, another encodes queries, and relevance is the inner product of the two vectors (§2.2). The paper initialises it from a DPR model already trained to find passages that answer TriviaQA and Natural Questions, and uses that same model to build the index.

Finding the top k passages with the highest inner product is a maximum inner product search (MIPS) problem, which can be solved approximately in sub-linear time. The index uses FAISS with a Hierarchical Navigable Small World approximation (§3).

The generator is BART-large, a pre-trained sequence-to-sequence transformer. Section 2.3 gives it 400M parameters; the appendix counts 406M trainable parameters for BART-large and 110M for each BERT-base encoder. To give BART a retrieved passage, the paper simply concatenates it with the question.

The retrieved passage is treated as a hidden (latent) variable: the model is never told which passage is right, so it sums over the top k it found. The paper defines two ways of doing that sum (§2.1).

RAG-Sequence uses the same passage for the whole answer. It computes the probability of the full answer under each passage, then takes a weighted average.

RAG-Token moves the sum inside the product: each output token can be drawn from a different passage. A Jeopardy clue that combines two facts from two articles is the paper’s example of where that helps.

Training uses pairs of inputs and outputs only: a question and its answer, a Jeopardy answer and its clue, a claim and its label. There is no direct supervision of which document should be retrieved. The objective is to minimise the negative marginal log-likelihood of each target, using Adam (§2.4).

Only two of the three networks are updated: the query encoder and BART. The document encoder is frozen, and with it the index. Updating it would mean re-encoding and re-indexing every passage periodically, as REALM does during pre-training; the authors report they “do not find this step necessary for strong performance”. This is the decision that makes the index swappable later.

Decoding differs between the two variants (§2.5). RAG-Token is a standard sequence-to-sequence model with a modified token probability, so an ordinary beam search works. RAG-Sequence is harder: the probability of an answer does not break into per-token pieces. The paper runs beam search once per document and then scores every candidate under every document (“Thorough Decoding”), or skips the extra forward passes by assuming a candidate that did not appear in a document’s beam has near-zero probability there (“Fast Decoding”).

Practical details from the appendices: models were trained with Fairseq in mixed precision across eight 32GB NVIDIA V100 GPUs, though the authors note training and inference can run on one GPU. The index vectors sat in CPU memory, about 100 GB for all of Wikipedia, reduced to 36GB with FAISS compression. For open-domain QA test scores, RAG-Token used 15 retrieved documents and RAG-Sequence 50 with Thorough Decoding; for generation tasks both used 10 documents and a beam size of four (Appendices A and C).

Every experiment uses the same knowledge source: the December 2018 Wikipedia dump, split into disjoint 100-word chunks, 21M documents in total (§3). During training the model retrieves the top 5 or top 10 documents. The tasks span four kinds of knowledge-intensive work:

Table 1 compares RAG with “closed-book” models that answer from their weights alone (T5-11B, with and without salient span masking) and with “open-book” retrieve-and-extract systems (REALM, DPR). Both RAG variants beat every closed-book model, and RAG set new best scores on Natural Questions, WebQuestions and CuratedTrec. On TriviaQA’s standard open test set, DPR stayed ahead.

Two details in §4.1 matter more than the headline. First, RAG can produce a correct answer even when no retrieved document contains it word for word: on Natural Questions it scores 11.8% on such cases, where an extractive model would score 0%. Second, the comparison with size is stark. The appendix notes that T5-large, at 770M parameters the closest T5 model in size to RAG, scores 28.9 exact match on Natural Questions, against 44.5 for RAG-Sequence with 626M trainable parameters.

On MS-MARCO, RAG-Sequence beats BART by 2.6 BLEU and 2.6 ROUGE-L points. It does not reach the state of the art, but those systems are given the gold passages that contain the answer, and many MS-MARCO questions cannot be answered from Wikipedia at all (§4.2). On Jeopardy, RAG-Token is best on both metrics. On FEVER, RAG gets within 4.3% of pipeline systems that are trained with supervision about which evidence to retrieve; RAG gets no such supervision.

Automatic metrics reward word overlap, not truth. So the authors asked people to compare 452 pairs of Jeopardy clues, one from BART and one from RAG-Token, for factuality and for specificity (§4.3).

FEVER comes with human-annotated gold evidence, which lets the authors check retrieval directly. The top retrieved document came from a gold article in 71% of cases, and a gold article was among the top 10 retrieved in 90% of cases (§4.4).

The experiment that best explains why RAG matters outside research is the shortest in the paper. The authors built a second index from a December 2016 Wikipedia dump, listed 82 world leaders who changed between 2016 and 2018, and asked the Natural Questions RAG model “Who is {position}?” for each, once with each index (§4.5).

With matching index and year the model was right 70% of the time (2016) and 68% (2018). With the wrong index, accuracy fell to 12% and 4%. The model weights were identical in all four runs. The answers followed the documents, which is the point: the authors conclude that RAG’s world knowledge can be updated “by simply replacing its non-parametric memory”.

Table 6 reports development-set scores for three versions of each model: with BM25 word-overlap search in place of DPR, with the retriever frozen, and with the full fine-tuned retriever.

Fine-tuning the retriever helps on every task, and most on open-domain QA. BM25 is far behind on question answering but is the best of the three on FEVER, whose claims are mostly about named entities and suit word overlap (§4.5). That result is worth remembering for any business system that has to match product codes, names or reference numbers.

On the number of documents: models trained with 5 or 10 retrieved passages performed about the same. At test time, retrieving more passages steadily improved RAG-Sequence on Natural Questions, while RAG-Token peaked at 10 (Figure 3 of the paper).

The paper’s own broader-impact section is direct about one limit: Wikipedia, “or any potential external knowledge source, will probably never be entirely factual and completely devoid of bias”. A model that answers from documents inherits their errors. The rest of these limits follow from the setup.

Most “answer from our documents” assistants follow the shape this paper made standard: search first, then write the answer from what was found. Three results translate directly.

Updating is an editing job, not a training job. The hot-swap test shows answers following the index. For a company, keeping the assistant current means keeping the documents current and re-indexing them.

Sources can be shown. Because each answer is conditioned on retrieved passages, a system can show which ones it used, and a person can check them. That is what makes it usable for policy, support and internal knowledge.

Keyword search still has a place. BM25 beating learned retrieval on FEVER is the paper’s own evidence that exact-term matching matters for names and identifiers. Most practical systems combine the two.