Pruning Weights and Channels

Reduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.

Intermediate · Undergraduate mathematics · about 20 minutes

Reading map and next steps

Intuition

The idea in plain words, before the symbols.

Pruning sets weights to zero, or removes whole structures such as channels and attention heads, while trying to keep the model's accuracy. We compare unstructured and structured masks, then use one small layer to show that two masks removing the same number of weights can reverse their order by output error when only the input changes. The figure separates reduced storage from a speedup, which needs kernels that skip zeros or smaller dense tensor shapes. Removing a weight does not establish that it was unnecessary for every input.

Two kinds of pruning behave very differently:

Unstructured pruning sets individual weights to zero. It can reach high sparsity, but it only runs faster if the hardware and software have kernels that skip the zeros (Han et al., 2016).
Structured pruning removes whole channels, heads or blocks. It usually removes fewer parameters at the same accuracy, but because it shrinks the dense tensor shapes, it is more likely to give real speedups on standard kernels (Li et al., 2017).

Between the two sit semi-structured patterns such as 2:4 sparsity, where two of every four consecutive weights are zero; NVIDIA's Ampere GPUs run this pattern at up to twice the dense math throughput (Mishra et al., 2021).

A simple rule, "remove the smallest weights", works surprisingly well: in a large comparison, magnitude pruning matched or beat more complex methods (Gale et al., 2019). It is not optimal, because a weight's size does not measure how much the loss depends on it. Methods that use approximate second-order information can do better: SparseGPT prunes the 175- and 176-billion-parameter OPT and BLOOM language models to 50% sparsity in one shot, without retraining and with little loss in accuracy (Frantar and Alistarh, 2023).

Counting the removed weights, or adding up their squares, does not say how much a layer's output changes: that depends on the input the layer receives. The practice after the demo builds a small example in which two masks with the same budget swap places when only the input changes.

Study prompts for this section: Intuition

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

ExploreChange one control at a time and name what responds.An optional observation guide follows the demo.

Explore Pruning Weights and Channels

Loading the interactive demo…

Observation guide

Optional and ungraded. Opening a guide saves nothing.

Choose what to inspect in Pruning Weights and Channels, then open its guide.

The demo below asks you to predict which pruning pattern actually turns into a speedup before it shows the mask and the deployment numbers. The point is that unstructured zeros reduce storage, and reduce computation only when sparse kernels skip them, while structured pruning changes the dense matrix shape and so maps to ordinary GPU speedups more directly.

Study prompts for this section: Demo

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Practice

Two masks remove the same number of weights. Predict which one changes the output more, check each number, then change only the input and predict again.

Loading the exercise…

Mathematics

Definitions, assumptions and the derivation.

Equation 1 Equation 2

Magnitude pruning (a common baseline)

Let $W$ be a weight matrix and let $\theta$ be a threshold. Define a mask:

M_{ij}=\mathbf 1[|W_{ij}|>\theta],\qquad W_{\text{pruned}}=W\odot M.

Structured pruning (remove units)

Let $W$ have one row per output unit and one column per input unit. If $S$ is the set of input units (columns), or of attention heads, that you keep, structured pruning changes the layer shape:

W_{\text{pruned}} = W[:,\,S].

This changes the model shape, which is why it can turn into speedups on ordinary dense kernels more directly. Each input unit is an output of the previous layer, so removing it also removes the matching row of the previous layer's weights. The figure below prunes the other way, removing whole rows (output units) of $W$ , which likewise removes the matching columns of the next layer.

Weight error and output error

Write $W_M=W\odot M$ for the pruned matrix, where $M_{ij}=1$ keeps an entry and $M_{ij}=0$ removes it. Two numbers are used to judge a mask. The removed-weight error $E_W(M)=\|W-W_M\|_F^2$ adds the squares of the removed entries. The output error $E_y(M;x)=\|Wx-W_Mx\|_2^2$ compares the layer's outputs on an input $x$ .

The first does not depend on the input, and it bounds the second for every input: $\|(W-W_M)x\|_2\le\|W-W_M\|_2\,\|x\|_2\le\|W-W_M\|_F\,\|x\|_2$ , so $E_y(M;x)\le E_W(M)\,\|x\|_2^2$ . A bound is not a ranking. A mask with the smaller $E_W$ can have the larger $E_y$ on a particular input.

The practice shows this with the $4\times4$ matrix whose rows are $(4, 0.8, 0.7, 0.6)$ , $(0.5, 3, 0.4, 0.3)$ , $(1.2, 1.2, 0.1, 0.1)$ and $(0.25, 0.25, 1, 1)$ , pruned in two ways that each remove eight of the sixteen entries. Magnitude pruning removes the eight smallest magnitudes, $0.1, 0.1, 0.25, 0.25, 0.3, 0.4, 0.5, 0.6$ ; the next is $0.7$ , so there is no tie at the cut. Row pruning removes the two rows with the smallest L1 norm: the row sums are $6.1, 4.2, 2.6, 2.5$ , so rows 3 and 4 go. Removed entries are set to zero, so both pruned layers keep four outputs; removing the rows from a real layer would also shorten the next layer's input.

The removed-weight errors are $E_W=1.005$ for the magnitude mask and $5.025$ for the row mask. On $x=(1,1,1,1)$ the output errors are $2.09$ and $13.01$ : the magnitude mask does less damage by both measures. Changing only the input, to $x'=(1,-1,1,-1)$ , reverses the output ranking: the errors become $0.72$ and $0$ . Rows 3 and 4 of $W$ give exactly zero on $x'$ , because their entries cancel in pairs, so removing them changes nothing for that input, while the magnitude mask still alters rows 1 and 2. The removed-weight errors stay at $1.005$ and $5.025$ .

This is one matrix and two inputs. It shows that removed-weight error does not rank masks by their effect on a given input; it does not show that row pruning is better. What matters in practice is the layer's actual inputs, which is why SparseGPT chooses the weights to remove, and adjusts the ones it keeps, so that the layer's outputs on a set of calibration inputs stay close to the dense layer's (Frantar and Alistarh, 2023).

Study prompts for this section: Mathematics

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import numpy as np rng = np.random.default_rng(0) W = rng.normal(size=(256, 256)).astype(np.f...python

import numpy as np

rng = np.random.default_rng(0)
W = rng.normal(size=(256, 256)).astype(np.float32)

sparsity = 0.7  # prune 70% of weights by magnitude
th = np.quantile(np.abs(W), sparsity)
M = (np.abs(W) > th).astype(np.float32)
W_pruned = W * M

print("target sparsity:", sparsity)
print("actual sparsity:", round(float((M == 0).mean()), 3))
print("L2 norm ratio ||W_pruned||/||W||:", round(float(np.linalg.norm(W_pruned) / np.linalg.norm(W)), 3))

The second block computes the worked example from the Mathematics section: both masks, both errors on both inputs, and the bound $E_y\le E_W\|x\|_2^2$ .

import numpy as np

W = np.array([[4, 0.8, 0.7, 0.6],
              [0.5, 3, 0.4, 0.3],
              [1.2, 1.2, 0.1, 0.1],
              [0.25, 0.25, 1, 1]])

# Magnitude mask: remove the 8 smallest |W_ij| (no tie at the cut: 0.6 < 0.7).
keep_mag = np.ones(16, dtype=bool)
keep_mag[np.argsort(np.abs(W), axis=None, kind="stable")[:8]] = False
keep_mag = keep_mag.reshape(4, 4)
# Row mask: remove the 2 rows with the smallest L1 norm, rows 3 and 4.
keep_row = np.ones((4, 4), dtype=bool)
keep_row[np.argsort(np.abs(W).sum(axis=1), kind="stable")[:2]] = False

def errors(keep, x):
    removed = np.where(keep, 0.0, W)            # W - W_M: the entries the mask sets to zero
    e_w = float((removed ** 2).sum())           # removed-weight error, the same for every x
    e_y = float(((removed @ x) ** 2).sum())     # output error on this x
    assert e_y <= e_w * float(x @ x) + 1e-12    # the bound E_y <= E_W ||x||^2
    return round(e_w, 6), round(e_y, 6)

for x in ([1, 1, 1, 1], [1, -1, 1, -1]):
    x = np.array(x, dtype=float)
    print("x =", x, "magnitude (E_W, E_y):", errors(keep_mag, x), "row (E_W, E_y):", errors(keep_row, x))

Study prompts for this section: Code

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Focus on

In focus

Concept: Pruning Weights and Channels

What is the smallest example of Pruning Weights and Channels you can work through by hand, and what does it show?

BeforeEfficiency: Quantization, Distillation, LoRA & Sparse MoENextKnowledge Distillation: Learning from Teachers

DetailsEfficiency

About this itemEfficiency

5 sources citedChecking this browser

Reference id

concept:efficiency/pruning

Math2 equationsM_{ij}=\mathbf 1[|W_{ij}|>\theta],\qquad W_{pruned}=W\odot M.CodePruning Weights and Channels code example 1assert e_y <= e_w * float(x @ x) + 1e-12 # the bound E_y <= E_W ||x||^2 DemoPruning Weights and Channels interactive demoChange one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.

What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

Reduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.

Choose what to look for (no demo yet)01 / Intuition

Illustration of a dense weight grid being pruned into a smaller sparse structure.

What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Pruning Weights and Channels.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

Reduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.

4 of 4 sections writtenNo live demo yet

Visual cueWhich part of the picture should you look at first?

Highlight size2/4

Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Pruning Weights and Channels.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.

paper · 2016Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman CodingHan, Mao, and Dally

Why it is cited

Prunes connections whose weights fall below a threshold, retrains the rest, stores the sparse result compactly and benchmarks sparse-kernel speed and energy; pruning alone cuts the number of connections by 9–13× (ICLR 2016).

Open source

paper · 2017Pruning Filters for Efficient ConvNetsLi et al.

Why it is cited

Removes whole convolutional filters and their feature maps, which leaves smaller dense layers that run on standard dense BLAS libraries without sparse kernels (ICLR 2017).

Open source

paper · 2019The State of Sparsity in Deep Neural NetworksGale, Elsen, and Hooker

Why it is cited

Compares sparsification methods at scale and finds that simple magnitude pruning achieves comparable or better results than more complex techniques (arXiv preprint).

Open source

paper · 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-ShotFrantar and Alistarh

Why it is cited

Prunes OPT-175B and BLOOM-176B to 50–60% unstructured sparsity in one shot, without retraining and with little loss in accuracy, and extends to the semi-structured 2:4 and 4:8 patterns (ICML 2023).

Open source

paper · 2021Accelerating Sparse Deep Neural NetworksMishra et al.

Why it is cited

Describes the sparse Tensor Cores of NVIDIA Ampere GPUs, which exploit a 2:4 pattern (two zeros in every four weights) to double the math throughput of dense matrix units (arXiv preprint).

Open source

What each claim rests on1 claim, each with its sources, where it appears on this page, and its caveats.

Claims and sources

Reduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.

Checked against the cited passages

The checks were made by this site, not by an independent reviewer.

Pruning applies a sparsity mask, often chosen with a magnitude threshold. Unstructured zeros can shrink storage but do not by themselves speed up dense kernels, while removing whole filters or channels changes tensor shapes and can speed up ordinary dense kernels.

Sources

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman CodingPrunes connections whose weights fall below a threshold, retrains the rest, stores the sparse result compactly and benchmarks sparse-kernel speed a...Pruning Filters for Efficient ConvNetsRemoves whole convolutional filters and their feature maps, which leaves smaller dense layers that run on standard dense BLAS libraries without spa...

What the sources support

Han et al. prune small-weight connections below a threshold, retrain remaining sparse connections, store them as CSR/CSC, and benchmark sparse matrix-vector kernels. Li et al. show filter pruning removes who...

Where to see it on this page

Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding Pruning Filters for Efficient ConvNets Equation 1 Equation 2 Code example 1 Demo

Equation 1

M_{ij}=\mathbf 1[|W_{ij}|>\theta],\qquad W_{\text{pruned}}=W\odot M.

Equation 2

W_{\text{pruned}} = W[:,\,S].

Code example 1import numpy as np rng = np.random.default_rng(0) W = rng.normal(size=(256, 256)).astype(np.f...

Caveat

This covers the mask and the difference between storage savings and speed. Compression ratios, retraining schedules, speedups on current hardware and accuracy after pruning vary by model and method; none is...

StatusChecked against the cited passages.

Han et al. support thresholded unstructured pruning of connections, masks, compressed sparse storage (CSR/CSC) and sparse-kernel benchmarks. Li et al. support removing filters and feature maps, which avoids irregular sparsity and leads to smaller dense BLAS operations.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Pruning Weights and Channels

Try the idea in your own words

Reduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.

Concept · Selected for practice

Pruning Weights and Channels

What it rests on: Sources: Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding; Pruning Filters for Efficient Co...

Context and links

Efficiency

Choose a task

Explain the mechanism

For Pruning Weights and Channels: What is the smallest example of Pruning Weights and Channels you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

Your answerNo answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

Where am I stuck? (optional)

Your own description, not an automatic diagnosis

Choose one, or leave this unspecified. Select it again to clear it.

Take your draft to a feedback conversation

No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

Write an attempt before copying a feedback prompt.

A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.

Ask about this pageClose

ConceptPruning Weights and ChannelsSources: Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding; Pruning Filters for Efficient Co...assert e_y <= e_w * float(x @ x) + 1e-12 # the bound E_y <= E_W ||x||^2

Your question

Choose what your question is about

Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptEfficiency

Pruning Weights and Channels

Starting question

What is the smallest example of Pruning Weights and Channels you can work through by hand, and what does it show?

SourcesCheck the 5 cited sources listed under Sources.Notes can be saved for this item

Points of view on this item

Fixed prompts generated from this item. They are not comments from people or independent reviews.

Learner’s first stepAsk what would make "Pruning Weights and Channels" feel predictable rather than familiar.

Assumption

The 5 cited sources must support this exact item, not just the surrounding topic.

Source-checking summary

Connect the definition to one equation, piece of code or demo before widening the discussion.

Proposed experiment

Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

Next action

You can state the mechanism in your own words

Evidence4 checks

ObservationChecking for a saved observation

ActionReady for one action

PromptLearner prompt ready to copy

Go to this item

01ObservationChecking what this browser has saved

02EvidenceChecking for a saved observation

03SourcesCheck the 5 cited sources listed under Sources.

04Next stepSave one next action

Local action draftNo local draft saved yetOpen when you are ready to write one next action

Local action draft

This draft stays in this browser and is attached to this item only.

Draft noteNext action

No local draft saved.

Evidence to inspect

What the 5 cited sources say about this exact item
The definition, its prerequisites, and a contrasting concept
The equation or code that makes the concept concrete
One demo state that shows what stays the same as the inputs change

What would resolve this

You can state the mechanism in your own words
You can name the prerequisite that would clear up the confusion
You can predict how the result changes when one input changes

Reference id

concept/concept-notebook/efficiency/pruning
concept:efficiency/pruning

Return to reading

Reading map

In this concept

Change one control at a time and name what responds.

Try the demo

Continue the curriculum

Part VI · unit 6 of 13

Knowledge Distillation: Learning from Teachers

Connections and sources

Domain

Efficiency

efficiency
pruning
sparsity

Prerequisites

Leads to

Quantization: Compressing Models to Integers

Knowledge Distillation: Learning from Teachers

Study aid

Ask an assistant about this notebook

Compose a question with this notebook as context, then copy it into an AI assistant you already use.

Compose a question

Your question

GoalComfortStyleStuck on

Copy-only tool: this panel composes a prompt from this page and your choices above, for an AI assistant you already use. Nothing you enter here is sent from this page.