Instruction tuning fine-tunes a pretrained language model, which continues text, on examples of requests and responses, so that it responds to the prompt as a request. Zero-shot FLAN outperformed zero-shot 175B GPT-3 on 20 of the 25 tasks evaluated (Wei et al., 2022).
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.
Pick a concrete input and say what the equation should give before you read the math.
CheckCheck against the math and the papers.
Work through the math, then read the cited papers for the original argument.
GeneralizeName what stays the same.
Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.
ContinueFollow the connections.
Use the prerequisites and related concepts at the end of the page to fill the next gap.
Why It Matters for Modern Models
Instruction tuning fine-tunes a pretrained language model, which continues text, on examples of requests and responses, so that it responds to the prompt as a request. Zero-shot FLAN outperformed zero-shot 175B GPT-3 on 20 of the 25 tasks evaluated (Wei et al., 2022).
The benefit depends on the range of tasks and on model scale. In FLAN, held-out performance rose as more task clusters were added, but for models of 8 billion parameters and smaller instruction tuning hurt it (Wei et al., 2022). Flan-PaLM 540B, tuned on about 1,800 tasks including chain-of-thought examples, beat PaLM 540B by 9.4% on average (Chung et al., 2024).
In InstructGPT, supervised fine-tuning on human demonstrations was the first of three stages, before reward modeling and PPO; labelers preferred the outputs of the 1.3-billion-parameter InstructGPT model to those of 175-billion-parameter GPT-3 (Ouyang et al., 2022).
Questions to Think About
Why did instruction tuning help the largest FLAN models on unseen tasks but hurt models of 8 billion parameters and smaller (Wei et al., 2022)?
Why does adding chain-of-thought examples to the tuning mix improve step-by-step reasoning, and what happens to that ability when they are left out (Chung et al., 2024)?
Core Math
Key Equation
L(θ)=−t∑logpθ(yt∣x,y<t)
Instruction tuning fine-tunes a pretrained language model on examples made of an instruction x, sometimes with an input, and a desired response y. The loss is the usual next-token cross-entropy, usually counted on the response tokens only:
L(θ)=−t=1∑∣y∣logpθ(yt∣x,y<t)
The data mixes many tasks, each rephrased with natural-language templates, for example "Summarize the following article:" followed by an article and its summary. FLAN fine-tuned a 137-billion-parameter model on more than 60 NLP datasets written this way and evaluated it zero-shot on groups of tasks held out from training (Wei et al., 2022).
In the RLHF pipeline the same step is called supervised fine-tuning (SFT), on demonstrations written by people; a reward model and reinforcement learning follow (Ouyang et al., 2022).