In-Context Learning: Learning Without Weight Updates

Why It Matters for Modern Models

In-context learning lets a language model take on a new task from a few examples in the prompt, without fine-tuning (Brown et al., 2020).
Changing behavior then means changing the prompt rather than the model, which makes it quick to try new tasks.
It explains why the choice, order and format of few-shot examples affect results, which is much of what prompt design is about.

Questions to Think About

What evidence links induction heads to in-context learning, and why does it fall short of showing that they are needed for every form of it?
When a model solves a task from demonstrations, how can we tell whether it learned the task from the examples or recognized one it had already seen in training?

Core Math

Key Equation

\hat{y} = \arg\max_y p_\theta(y \mid \text{examples}, x_{\text{query}})

In-context learning is adapting to a task from examples given in the prompt, with no change to the weights.

Given demonstrations $(x_1, y_1), \ldots, (x_k, y_k)$ and a query $x_{k+1}$ :

\hat{y}_{k+1} = \arg\max_y p_\theta(y \mid x_1, y_1, \ldots, x_k, y_k, x_{k+1})

The parameters $\theta$ stay fixed; the model only conditions on the demonstrations. Brown et al. (2020) found that the benefit of adding demonstrations grows with model size, but the ability itself does not require a large model: small transformers trained on suitable data learn in context too, for example fitting new linear functions from a few examples (Garg et al., 2022).

One proposed mechanism: implicit gradient descent. For linear regression, a single linear self-attention layer can compute the result of one gradient-descent step on the in-context examples (von Oswald et al., 2023):

W_{\text{updated}} = W + \eta \sum_{i=1}^{k} (y_i - Wx_i)\,x_i^\top

This was shown for small models trained on regression tasks. Whether large pretrained language models learn in context this way is disputed (Shen et al., 2024).

Another: induction heads. These attention heads find an earlier occurrence of the current token and copy what followed it. Olsson et al. (2022) present evidence that they are a major mechanism of in-context learning, strongest for small attention-only models.