Infinite-context architectures handle inputs too long for ordinary attention in one of two ways: they keep a fixed-size compressed memory of earlier segments, or they split exact attention across many devices. Infini-attention let a 1-billion-parameter model, fine-tuned on 5,000-token inputs, retrieve a passkey hidden in 1 million tokens, and an 8-billion-parameter model summarize books of 500,000 tokens (Munkhdalai et al., 2024).
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Infinite-context architectures handle inputs too long for ordinary attention in one of two ways: they keep a fixed-size compressed memory of earlier segments, or they split exact attention across many devices. Infini-attention let a 1-billion-parameter model, fine-tuned on 5,000-token inputs, retrieve a passkey hidden in 1 million tokens, and an 8-billion-parameter model summarize books of 500,000 tokens (Munkhdalai et al., 2024).
Ring Attention trains and runs exact attention on sequences up to the number of devices times longer than earlier memory-efficient transformers could handle (Liu, Zaharia & Abbeel, 2024).
A long window is not the same as using it well: models often do worst when the relevant information sits in the middle of a long input (Liu et al., 2024), and on the RULER benchmark only about half of the models that claim contexts of 32,000 tokens or more performed satisfactorily at 32,000 (Hsieh et al., 2024).
Questions to Think About
Ring Attention keeps attention exact by spreading it over devices; why does that bound the memory on each device but not the total computation?
Why might a model that can retrieve a single hidden passkey from a million tokens still fail at tasks that require combining many facts spread across the input (Hsieh et al., 2024)?
Core Math
Key Equation
Ms=Ms−1+σ(K)⊤V
Standard attention over n tokens,
Attn(Q,K,V)=softmax(dQK⊤)V,
costs O(n2) computation, and during generation the cache of keys and values grows linearly with n. "Infinite-context" methods attack these costs in two different ways.
Compressive memory. Infini-attention processes the input in segments. Within a segment it uses ordinary local attention; across segments it keeps a fixed-size memory matrix M, updated with each segment's keys and values as in linear attention (Munkhdalai et al., 2024):
Ms=Ms−1+σ(K)⊤V,Amem=σ(Q)zs−1σ(Q)Ms−1
where σ is ELU + 1 and zs accumulates σ(K) for normalization. A learned gate mixes Amem with the local attention output. Memory and compute per segment stay bounded however long the input is, but the memory is a lossy summary of the past.
Distributed exact attention. Ring Attention splits a long sequence into blocks held on different devices, and passes key-value blocks around a ring of devices while each device computes blockwise attention, overlapping communication with computation (Liu, Zaharia & Abbeel, 2024). Attention stays exact and the maximum length grows with the number of devices, but the total computation is still quadratic in n.
Key Papers
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention