Retrieval lets a language model use information that is newer than its training data, or was never in it, and lets that information be updated without retraining.
Concept 43 of 100RAGRepresentations · Study phase 9: Advanced architectures & generation
Question
When can a smaller model with retrieval match a larger one without it, as RETRO performed comparably to GPT-3 on the Pile with 25 times fewer parameters (Borgeaud et al., 2022)?
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Retrieval lets a language model use information that is newer than its training data, or was never in it, and lets that information be updated without retraining.
Dense retrieval, which finds passages by the similarity of learned embeddings, outperformed keyword-based BM25 retrieval on open-domain question answering (Karpukhin et al., 2020).
Retrieval separates what a model has memorized from what it can look up, which makes its sources inspectable and its knowledge editable.
Questions to Think About
Why can irrelevant retrieved passages make answers worse than retrieving nothing, so that retrieval quality becomes the bottleneck?
Why do language models use information at the start or end of a long context more reliably than information in the middle, and what does that imply for how retrieved passages are ordered (Liu et al., 2024)?
Core Math
Key Equation
p(y∣x)≈d∈top-k∑p(d∣x)p(y∣x,d)
Retrieval-augmented generation (RAG) retrieves documents relevant to the input and conditions the generator on them.
Retrieval scores documents by embedding similarity:
p(d∣x)∝exp(Ed(d)⊤Eq(x))
where Eq and Ed are query and document encoders. In Lewis et al. (2020) they are the two BERT-based encoders of dense passage retrieval (Karpukhin et al., 2020), and the top k documents are found by maximum inner-product search.
Marginalizing over retrieved documents (the RAG-Sequence model):
p(y∣x)≈d∈top-k∑p(d∣x)⋅p(y∣x,d)
with p(d∣x) renormalized over the top k documents.
Generating from concatenated context. Many current systems skip the marginalization and place the retrieved passages directly in the prompt:
p(y∣x,d1,…,dk)=t∏p(yt∣y<t,x,d1,…,dk)
Key Papers
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks