Quantization: Compressing Models to Integers

Uniform quantization (per-tensor)

Given a tensor of weights with minimum $w_{\min}$ and maximum $w_{\max}$, choose one step size for the whole tensor and round onto integer levels:

Here $b$ is the number of bits (8 for INT8, 4 for INT4), an integer from 1 to 16 in the code below, and $w_{\min}$ and $w_{\max}$ are the smallest and largest entries of a nonempty tensor of finite float32 values. This is an asymmetric (min–max) quantizer: the offset $w_{\min}$ shifts the range, so integer level $0$ stands for $w_{\min}$ and level $2^b-1$ for $w_{\max}$. Integer-only schemes round the offset to an integer zero point $Z$ and write $r = S(q - Z)$, so that the real value 0 is represented exactly (Jacob et al., 2018).

When every entry is equal (a single value, or a row of zeros), $w_{\max}=w_{\min}$ and the quotient $(w-w_{\min})/\Delta$ would be $0/0$. One level is then enough: $w_q=0$ and $\hat w=w_{\min}$ reproduce the tensor exactly. This branch is a definition chosen for that case, not a measured result.

The code below computes this definition in floating point. Three details matter:

• Rounding. It computes the quotient $(w-w_{\min})/\Delta$ in float64 and rounds it to the nearest integer, sending exact halves to the even integer, as np.round does. That rounds the computed quotient, which is not always the exact real one. At $b=2$, the float32 entries $[-1,\,-10^{-20},\,1]$ give a computed quotient of exactly $1.5$ for the middle entry, so it goes to level $2$ ($\hat w=1/3$), although the exact nearest level is $1$ ($\hat w=-1/3$).
• Endpoints. It keeps every reconstructed level inside $[w_{\min},w_{\max}]$ and sets the top level to $w_{\max}$ exactly. Without that step, for $[-f_{\max},\,1]$, where $f_{\max}$ is the largest float32, the sum $w_{\min}+(2^b-1)\Delta$ loses the $1$ to cancellation.
• Underflow. The final conversion to float32 rounds to nearest, which can turn a tiny level into a subnormal number or zero. This narrowing is the only step where the code allows floating-point underflow.

Finer-grained scaling: per-channel / row-wise

Instead of one scale for the whole matrix, use one scale per output channel/row. For signed symmetric quantization:

This usually improves quality because different channels have different ranges of values. Here $b$ must be an integer with $b\ge 2$: at $b=1$, $q_{\max}=0$ and there is no nonzero level, which is narrower than the affine rule's $1\le b\le 16$. Symmetric quantization keeps the real value 0 at integer level $0$ and uses the levels $-q_{\max},\dots,q_{\max}$, leaving $-2^{b-1}$ unused. An all-zero row has $s_i=0$; by the same convention as above, $Q_{i,j}=0$ and $\hat W_{i,j}=0$ exactly, with no division.

The two formulas are different quantizers, not two spellings of one. At $b=2$, the tensor $[1,2,3]$ reconstructs as $[1,\,7/3,\,3]\approx[1,\,2.33,\,3]$ under the affine rule and as $[0,\,3,\,3]$ under the symmetric one. Dettmers et al. (2022) describe both families for 8-bit inference, absmax (symmetric) and zeropoint (asymmetric) quantization; these formulas are teaching versions, not the whole LLM.int8() method.

Clipping and rounding

A scale can cause two different errors. A value whose rounded code falls outside $[-q_{\max},q_{\max}]$ is clipped: it is reconstructed at the end of the range, however far beyond the range it lies. A value inside the range is rounded to the nearest level, so it is off by at most half a step, $s_i/2$. The absmax scale above never clips, because the largest magnitude maps exactly to $q_{\max}$; the price is that one large value sets the step for every other value that shares its scale. A smaller scale, such as one chosen before an outlier is seen, keeps a fine step for the ordinary values and clips the outlier instead. The practice after the demo works through both errors on a $2\times4$ example.

Per-row scales let each row pay only for its own range. When every row has the same largest magnitude, each row's absmax scale equals the shared one, so per-row and shared scales give the same result. Group-wise scales, described next, can still help then. If both rows are $(-0.7,-0.3,0.3,7)$, both row scales are $1$ and the root-mean-square error is $\sqrt{0.54/8}\approx0.26$; with one scale for each group of two weights, $0.1$ and $1$, it falls to $\sqrt{0.18/8}=0.15$.

Group-wise scales

At 4 bits even one scale per row is often too coarse. Group-wise quantization splits each row into groups of $g$ consecutive weights, commonly 64 or 128, and gives each group its own scale. The scales cost memory too: a 16-bit scale for every 128 weights adds $16/128 = 0.125$ bits per weight, so a "4-bit" model stores about 4.125 bits per weight, or more if each group also has a zero point.

Beyond rounding: GPTQ, AWQ and SmoothQuant

Rounding each weight to its nearest level is the simplest choice, not the most accurate one. Three widely used methods for large language models do better with a little calibration data:

• GPTQ (Frantar et al., 2023) quantizes a layer's weights one column at a time, aiming to keep the layer's outputs on calibration inputs $X$ (one column per calibration token) close to the original, $\lVert WX - \hat WX\rVert^2$. After each column it adjusts the columns not yet quantized to compensate for the rounding error, using approximate second-order information (the inverse of $H = 2XX^\top$). It brought 175-billion-parameter models to 3–4 bits per weight in about four GPU hours.
• AWQ (Lin et al., 2024) quantizes weights only. It observes that about 1% of weight channels matter most, and that they are the ones multiplying large activations, not the ones with the largest weights. It scales those channels up before quantization and divides the matching activations by the same factor, which leaves the product unchanged but reduces their relative rounding error. The scales are chosen from calibration activations, without backpropagation.
• SmoothQuant (Xiao et al., 2023) targets 8-bit weights and activations. Activation outliers sit in a few channels, which makes activations hard to quantize while weights are easy. Writing the layer as $Y = XW$, with one row of $X$ per token and one column per input channel $j$, it moves part of the difficulty into the weights with an equivalent rescaling, $Y = \big(X\,\mathrm{diag}(s)^{-1}\big)\big(\mathrm{diag}(s)\,W\big)$, where $s_j = \max_t |X_{t,j}|^{\alpha} / \max_k |W_{j,k}|^{1-\alpha}$ and $\alpha = 0.5$ is the default balance.