Modern CNN Block Mechanics

Let $x\in\mathbb R^{C\times H\times W}$ be the input activation to a CNN block. A dense $K\times K$ convolution from $C_{\text{in}}$ input channels to $C_{\text{out}}$ output channels uses

$$K^2C_{\text{in}}C_{\text{out}}$$

weights, ignoring bias and normalization. That single formula explains why so many modern blocks factorize or reroute the work.

Residual Bottleneck

A residual block writes

$$y=x+F(x)$$

when the shortcut has the same shape as the residual branch output. If the shape changes, a projection shortcut can map $x$ to the output shape. The key learning move is not merely "add a skip connection"; it is that the block can preserve an identity-like path while $F$ learns a correction.

A bottleneck residual branch often uses

$$C \xrightarrow{1\times1} C_b \xrightarrow{3\times3} C_b \xrightarrow{1\times1} C,$$

where $C_b<C$. Its branch parameter count is

$$CC_b + 9C_b^2 + C_bC.$$

For $C=256$ and $C_b=64$, this is

$$256\cdot64 + 9\cdot64^2 + 64\cdot256 = 69632.$$

Two dense $3\times3$ convolutions at width $256$ would use

$$2\cdot9\cdot256^2 = 1179648$$

weights. This is a simplified comparison, but it reveals the design pressure: keep the residual route, do expensive spatial mixing at a narrower width, and expand back to the block width.

Cardinality And Grouped Convolution

ResNeXt emphasizes cardinality: multiple grouped transformations can be aggregated inside a residual block. A grouped $K\times K$ convolution with $g$ equal groups uses

$$\frac{K^2C_{\text{in}}C_{\text{out}}}{g}$$

weights when channels divide evenly. For $K=3$, $C_{\text{in}}=C_{\text{out}}=256$, and $g=32$:

$$\frac{9\cdot256^2}{32}=18432.$$

That number is not the whole block cost, because practical designs also need channel mixing and surrounding $1\times1$ layers. The mechanism is still important: grouped transformations can increase the number of parallel transformation paths without paying a full dense spatial convolution for each path.

Depthwise Separable Spatial Mixing

MobileNet popularized depthwise separable convolution as an efficiency pattern. A depthwise $K\times K$ convolution applies one spatial filter per input channel, costing

$$K^2C_{\text{in}}.$$

A following pointwise $1\times1$ convolution mixes channels, costing

$$C_{\text{in}}C_{\text{out}}.$$

So the factorized cost is

$$K^2C_{\text{in}} + C_{\text{in}}C_{\text{out}}.$$

For $K=3$, $C_{\text{in}}=128$, and $C_{\text{out}}=256$, dense convolution uses $294912$ weights, while depthwise plus pointwise uses

$$9\cdot128 + 128\cdot256 = 33920.$$

Depthwise separability is a parameter and compute story, not a guarantee that every hardware kernel or accuracy target improves.

Channel Recalibration

A squeeze-and-excitation block makes channel weights depend on the current image. First it squeezes each channel by global average pooling:

$$z_c=\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W} x_{c,h,w}.$$

Then a small gating network produces channel scales

$$s=\sigma(W_2\delta(W_1z)),$$

and the original feature map is recalibrated by

$$\tilde{x}_{c,h,w}=s_cx_{c,h,w}.$$

With reduction ratio $r$, the two fully connected matrices use roughly

$$2\frac{C^2}{r}$$

weights. For $C=128$ and $r=16$, that is $2048$ weights. The gate is cheap compared with a dense $3\times3$ convolution at the same width, but it adds a specific kind of adaptivity: channel-wise recalibration. It is not Transformer attention over tokens.

Compound Scaling

EfficientNet asks a different question: once the block family is chosen, how should a model scale? A simplified compound-scaling rule writes

$$d=\alpha^\phi,\qquad w=\beta^\phi,\qquad r=\gamma^\phi,$$

where $d$ scales depth, $w$ scales channel width, and $r$ scales input resolution. Since dense convolution cost roughly scales with depth, width squared, and resolution squared, the approximate compute multiplier is

$$d\,w^2\,r^2.$$

The point is coordinated scaling. Wider, deeper, and higher-resolution models should not be treated as independent knobs when the compute budget is fixed.

ConvNeXt-Style Block

ConvNeXt revisits ConvNets after transformer-era design habits. A simplified ConvNeXt block can be read as

$$x \rightarrow \text{depthwise }7\times7 \rightarrow \text{LayerNorm} \rightarrow \text{pointwise expand} \rightarrow \text{GELU} \rightarrow \text{pointwise contract} \rightarrow \text{residual add}.$$

TorchVision's ConvNeXt block follows this pattern with a $7\times7$ depthwise convolution, LayerNorm, pointwise linear layers with a $4C$ hidden width, GELU, layer scale, stochastic depth, and residual addition.

For $C=96$, a dense $7\times7$ convolution would use

$$49\cdot96^2=451584$$

weights. A depthwise $7\times7$ plus $1\times1$ expansion and contraction uses

$$49\cdot96 + 96\cdot384 + 384\cdot96 = 78432.$$

This arithmetic explains why large depthwise kernels are affordable. It does not by itself explain ConvNeXt performance. The full design also includes stage widths, downsampling, normalization placement, activation placement, layer scale, stochastic depth, training recipe, and data regime.