LLM Systems

RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation

RAG retrieval is evidence triage: sparse and dense first-stage retrievers, chunking, reranking, and ranked metrics all fail in different ways.

status: reviewimportance: criticaldifficulty 4/5math: graduateread: 24mlive demo

Back to LLM Systems Next: Retrieval-Augmented Generation: External Memory for Generation (review)

Concept Structure

RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

3prerequisites

1next concepts

4related links

Learner Contract

What this page should let you do.

You are here becauseRAG retrieval is evidence triage: sparse and dense first-stage retrievers, chunking, reranking, and ranked metrics all fail in different ways.

This LLM Systems concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thiskNN and Nearest-Neighbor Geometry (review)

3 prerequisites listed; refresh them before leaning on the math or code.

kNN and Nearest-Neighbor Geometry (review)Representation Learning & Embedding Geometry Train/Dev/Test Splits, Cross-Validation, and Leakage

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextRetrieval-Augmented Generation: External Memory for Generation (review)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Retrieval-Augmented Generation: External Memory for Generation (review)

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims1/1 reviewed

Sources6 cited

Codeattached

Demolive

Reviewed2026-07-02

Updatedpage 2026-07-02

Learning item flow

4/4 sections readyAsk about this Research room

ConceptRAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunkin...LLM Systems EquationRAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunkin...Exact equation CodeRAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunkin...Exact runnable code DemoRAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunkin...Interactive demo ClaimRAG retrieval is a ranked-evidence problem: sparse BM25, dense embedd...Exact claim check SourceIntroduction to Information Retrieval: Okapi BM25Exact source

ConceptRAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and EvaluationLLM Systems

6 sources attachedLocal snapshot ready

concept:llm-systems/rag-retrieval-bm25-dense-reranking-chunking-eval

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Retrieval-augmented generation is often described as “chat with your documents.” That phrase is useful for a product demo, but it hides the part that determines whether the model sees good evidence.

Retrieval is an evidence-triage system.

For each query, the system must decide:

What units are retrievable? Sentences, passages, overlapping chunks, tables, code cells, or whole documents?
What first-stage scorer gives enough recall? Sparse keyword scoring, dense embeddings, or a hybrid?
Should a slower reranker look at the query and each candidate together?
Did the ranked list actually contain evidence, or did the generator merely sound confident afterward?

BM25 and dense retrieval fail differently. BM25 can love exact words and miss paraphrases. Dense retrieval can love semantic neighbors and pull in a persuasive distractor. Chunking can split the answer across boundaries, while an oversized chunk can bury the answer in unrelated context. A reranker can rescue a candidate set when the right evidence was retrieved somewhere, but it cannot rerank evidence that never arrived.

That is why retrieval evaluation should happen before answer generation. A learner should be able to look at a query, a small corpus, and a ranked list and ask: “Was the supporting evidence in the top $k$ ? At what rank? Is the cited span actually enough?”

The central habit is:

Treat RAG retrieval as a measurable ranked-evidence pipeline, not as a magic truth layer.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $\mathcal C=\{c_1,c_2,\ldots,c_N\},$ Equation 2 $C_K(q)=\operatorname{TopK}_{c_i\in\mathcal C} s_0(q,c_i).$

Let a corpus be split into chunks

\mathcal C=\{c_1,c_2,\ldots,c_N\},

where each chunk $c_i$ is the unit the retriever can return. A query is $q$ . A first-stage retriever assigns a score $s_0(q,c_i)$ and returns the top- $K$ candidate set

C_K(q)=\operatorname{TopK}_{c_i\in\mathcal C} s_0(q,c_i).

For a sparse lexical retriever, Okapi BM25 scores a document or chunk by summing term-level evidence. One common form is

s_{\mathrm{BM25}}(q,c) = \sum_{t\in q} \operatorname{IDF}(t) \frac{f(t,c)(k_1+1)} {f(t,c)+k_1\left(1-b+b\frac{|c|}{\operatorname{avgdl}}\right)}.

Here $t$ is a query term, $f(t,c)$ is the frequency of term $t$ in chunk $c$ , $|c|$ is the chunk length, $\operatorname{avgdl}$ is average chunk length, $k_1$ controls term-frequency saturation, and $b$ controls length normalization. The important shape is not just “keyword overlap.” BM25 rewards informative query terms, saturates repeated terms, and normalizes for chunk length.

For dense retrieval, an encoder maps the query and each chunk to vectors:

z_q=e_q(q),\qquad z_i=e_c(c_i).

A bi-encoder retriever can rank chunks by dot product or cosine similarity:

s_{\mathrm{dense}}(q,c_i)=z_q^\top z_i.

This is nearest-neighbor search in representation space. It can catch paraphrases, but its errors are geometric: a chunk can be close because it is topically similar, not because it contains the exact needed evidence.

A reranker usually starts from first-stage candidates and assigns a slower interaction score

r(q,c)=g([q;c]).

The final list is

R_M(q)=\operatorname{TopM}_{c\in C_K(q)} r(q,c).

The dependency on $C_K$ matters. If chunking or first-stage retrieval excludes the true support, the reranker cannot invent it.

For evaluation, let $G(q)\subseteq\mathcal C$ be the set of chunks judged relevant or citation-supporting. If $R_k(q)$ is the top- $k$ retrieved list, then a single-query recall-at- $k$ indicator is

\operatorname{Recall@}k(q)=\mathbf 1\left[G(q)\cap R_k(q)\ne \varnothing\right].

If the first relevant result appears at rank $\rho(q)$ , then

\operatorname{RR}(q)= \begin{cases} 1/\rho(q), & \text{if a relevant chunk is retrieved},\\ 0, & \text{otherwise.} \end{cases}

For graded relevance labels $\operatorname{rel}_j$ at rank $j$ ,

\operatorname{DCG@}k=\sum_{j=1}^k \frac{2^{\operatorname{rel}_j}-1}{\log_2(j+1)}, \qquad \operatorname{nDCG@}k=\frac{\operatorname{DCG@}k}{\operatorname{IDCG@}k}.

These are retrieval metrics. They do not say the generator will quote faithfully, avoid unsupported synthesis, or notice stale evidence. Citation support and answer correctness must be evaluated separately.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1import math, numpy as np chunks = [ ("A", "the theorem assumes gaussian initialization and in...python

import math, numpy as np

chunks = [
    ("A", "the theorem assumes gaussian initialization and infinite width", [0.86, 0.55], 1),
    ("B", "a blog says the theorem works for many neural networks", [0.92, 0.88], 0),
    ("C", "assumption notes for optimizer step size and clipping", [0.30, 0.20], 0),
    ("D", "older note claims finite width is enough without caveats", [0.78, 0.62], 0),
]
query = "what assumptions make the theorem valid"
q_vec = np.array([0.90, 0.70])
relevant = {doc_id for doc_id, _, _, rel in chunks if rel}

def bm25_like(text):
    q_terms, terms = set(query.split()), text.split()
    return sum(1 + math.log(1 + terms.count(t)) for t in q_terms & set(terms))

def dense(vec):
    v = np.array(vec)
    return float(q_vec @ v / (np.linalg.norm(q_vec) * np.linalg.norm(v)))

def rerank(doc_id, lexical, semantic):
    support_bonus = 0.55 if doc_id == "A" else 0.0
    stale_penalty = 0.45 if doc_id == "D" else 0.0
    return 0.35 * lexical + 0.65 * semantic + support_bonus - stale_penalty

rows = [(doc_id, bm25_like(text), dense(vec), rel) for doc_id, text, vec, rel in chunks]
scores = {
    "bm25": [(doc_id, lex, rel) for doc_id, lex, _, rel in rows],
    "dense": [(doc_id, sem, rel) for doc_id, _, sem, rel in rows],
    "rerank": [(doc_id, rerank(doc_id, lex, sem), rel) for doc_id, lex, sem, rel in rows],
}

for name, ranked in scores.items():
    ranked = sorted(ranked, key=lambda x: x[1], reverse=True)
    order = [doc_id for doc_id, _, _ in ranked]
    first_rel = next((i + 1 for i, x in enumerate(order) if x in relevant), None)
    recall3 = any(x in relevant for x in order[:3])
    mrr = 0 if first_rel is None else 1 / first_rel
    print(name, order, "Recall@3", recall3, "MRR", round(mrr, 3))

The point is not that these hand-written scores are a production retriever. The point is that retrieval has a ledger: one ranking can be keyword-driven, another semantic, and a reranker can only repair candidates that survived the earlier stage.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

The lab asks you to predict which retrieval stage will put the true supporting chunk at rank 1 before you inspect the ledger. Change the chunking profile and retrieval stage, commit to a hypothesis, then reveal the top-3 evidence, Recall@3, MRR, nDCG, and the failure note.

Use it as a small source-grounding ritual: before asking a generator to answer, decide whether the retriever has actually placed evidence where the generator can see it.

Live Concept Demo

Explore RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

RAG retrieval is evidence triage: sparse and dense first-stage retrievers, chunking, reranking, and ranked metrics all fail in different ways.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation should make visible.

Visual Inquiry

Make the image answer a mathematical question

RAG retrieval is evidence triage: sparse and dense first-stage retrievers, chunking, reranking, and ranked metrics all fail in different ways.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2008Introduction to Information Retrieval: Okapi BM25Manning, Raghavan, and Schutze

Supports the BM25 term-frequency, inverse-document-frequency, and length-normalization retrieval score used as the sparse retrieval baseline.

Open source

book · 2008Introduction to Information Retrieval: Evaluation of Ranked Retrieval ResultsManning, Raghavan, and Schutze

Supports ranked retrieval evaluation with precision-recall tradeoffs, mean average precision, and the need to evaluate ranking quality rather than only final answer quality.

Open source

paper · 2020Dense Passage Retrieval for Open-Domain Question AnsweringKarpukhin et al.

Supports bi-encoder dense retrieval, query/passage embeddings, inner-product ranking, and the contrast with sparse retrieval for open-domain QA.

Open source

paper · 2019Passage Re-ranking with BERTNogueira and Cho

Supports cross-encoder reranking over first-stage candidates by scoring query-passage pairs jointly.

Open source

paper · 2021BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval ModelsThakur et al.

Supports evaluating retrieval systems across heterogeneous tasks and warns against over-trusting one retrieval setting.

Open source

documentation · 2026LangChain Docs: Text SplittersLangChain

Supports the practical framing that chunking chooses the retrievable units and balances semantic coherence, retrieval granularity, and context length.

Open source

Claim Review

RAG retrieval is evidence triage: sparse and dense first-stage retrievers, chunking, reranking, and ranked metrics all fail in different ways.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources6 references

manning-2008-iir-bm25, manning-2008-iir-ranked-eval, karpukhin-2020-dpr, nogueira-2019-bert-reranking, thakur-2021-beir, langchain-2026-text-splitters

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedRAG retrieval is a ranked-evidence problem: sparse BM25, dense embedding retrieval, chunking, and reranking expose different failure modes, while Recall@k, MRR, nDCG, and citation support measure retrieval quality rather than guaranteeing generated-answer truth.Claim metadata: source checked

The lesson uses the sources for the retrieval mechanisms and metric vocabulary, then uses a finite toy corpus to make their failure modes visible before generation.

Sources: Introduction to Information Retrieval: Okapi BM25, Introduction to Information Retrieval: Evaluation of Ranked Retrieval Results, Dense Passage Retrieval for Open-Domain Question Answering, Passage Re-ranking with BERT, BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models, LangChain Docs: Text SplittersThe lab is a deterministic teaching witness, not a production vector database, exact BM25 implementation, learned embedding model, or evidence that retrieved context will be faithfully used by a generator.A bounded review summary is present; still check caveats and exact reference scope.

Checked Stanford IIR BM25/ranked-eval chapters, DPR, BERT reranking, BEIR, and LangChain text splitting. The central synthesis is supported: sparse and dense retrieval, chunking, reranking, and ranked metrics are separate evidence-quality levers; generated-answer truth remains a separate evaluation.

Reviewer: codex-local-source-review; reviewed 2026-07-02

source-span-manning-2008-iir-bm25 source-span-manning-2008-iir-ranked-eval source-span-karpukhin-2020-dpr source-span-nogueira-2019-bert-reranking Equation 1 Runnable code 1 Interactive demo

Source support candidates

book 2008Introduction to Information Retrieval: Okapi BM25

Supports the BM25 term-frequency, inverse-document-frequency, and length-normalization retrieval score used as the sparse retrieval baseline.

book 2008Introduction to Information Retrieval: Evaluation of Ranked Retrieval Results

Supports ranked retrieval evaluation with precision-recall tradeoffs, mean average precision, and the need to evaluate ranking quality rather than only final answer quality.

paper 2020Dense Passage Retrieval for Open-Domain Question Answering

Supports bi-encoder dense retrieval, query/passage embeddings, inner-product ranking, and the contrast with sparse retrieval for open-domain QA.

paper 2019Passage Re-ranking with BERT

Supports cross-encoder reranking over first-stage candidates by scoring query-passage pairs jointly.

Mechanism checks

Equation 1

\mathcal C=\{c_1,c_2,\ldots,c_N\},

Equation 2

C_K(q)=\operatorname{TopK}_{c_i\in\mathcal C} s_0(q,c_i).

Runnable code 1import math, numpy as np chunks = [ ("A", "the theorem assumes gaussian initialization and in...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

RAG retrieval is evidence triage: sparse and dense first-stage retrievers, chunking, reranking, and ranked metrics all fail in different ways.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptRAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and EvaluationLLM Systems

Runnable code comparisonRAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation...scores = {Prediction before revealRAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation...Manipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptLLM Systems

RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation

Attached question

What is the smallest example that makes RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation Selected item key: recorded for copy. Context: LLM Systems Page anchor: recorded for copy. Open question: What is the smallest example that makes RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/llm-systems/rag-retrieval-bm25-dense-reranking-chunking-eval
concept:llm-systems/rag-retrieval-bm25-dense-reranking-chunking-eval

Learning Map

Before / Now / Try / Next

BeforekNN and Nearest-Neighbor Geometry (review)

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextRetrieval-Augmented Generation: External Memory for Generation (review)

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

RAG retrieval is evidence triage: sparse and dense first-stage retrievers, chunking, reranking, and ranked metrics all fail in different ways.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: LLM Systems concept. Learning surface: RAG Retrieval Fundamentals: BM25, Dense Retrieval, Reranking, Chunking, and Evaluation. What this page says: RAG retrieval is evidence triage: sparse and dense first-stage retrievers, chunking, reranking, and ranked metrics all fail in different ways. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

LLM Systems

llm-systemsretrievalragbm25dense-retrievalrerankingevaluation

Prerequisites

kNN and Nearest-Neighbor Geometry (review)Representation Learning & Embedding Geometry Train/Dev/Test Splits, Cross-Validation, and Leakage

Leads To

Retrieval-Augmented Generation: External Memory for Generation (review)

Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization LLM Serving at Scale: Prefill, Decode & Continuous Batching Structured Decoding: Token Masks From Schema Automata Classification Metrics, Thresholds, and Calibration

Within this domain