CNN Architecture Families

Start with a plain stack of blocks. Let $x_l\in\mathbb R^{C_l\times H_l\times W_l}$ be the activation entering layer or block $l$. A plain CNN block can be written as

$$x_{l+1}=H_l(x_l),$$

where $H_l$ may include convolution, normalization, activation, pooling, or downsampling. The architecture question is the shape of $H_l$ and whether the block has extra routes around or into it.

Small Kernel Depth

VGG's central move is to replace one large spatial filter with repeated small filters. In the stride-1, no-dilation, appropriate-padding simplification, two $3\times3$ filters have the same nominal receptive-field width as one $5\times5$ filter. If input and output channel counts are both $C$, one $5\times5$ convolution uses

$$25C^2$$

weights. Two $3\times3$ convolutions have an effective $5\times5$ receptive field and use

$$2\cdot9C^2=18C^2$$

weights, while also inserting an extra nonlinearity between them. Three $3\times3$ convolutions similarly cover a $7\times7$ receptive field with $27C^2$ weights instead of $49C^2$.

This simplified count assumes equal channel width and ignores bias, normalization, and boundary effects. Its point is the design pressure: spend depth and nonlinearities more carefully than a single large dense spatial filter.

Multi-Scale Branches

An Inception-style block applies several branches to the same input and concatenates their channel outputs:

$$y=\operatorname{concat}(B_1(x), B_3(x), B_5(x), P(x)).$$

A direct $K\times K$ convolution from $C_{\text{in}}$ channels to $C_{\text{out}}$ channels costs

$$K^2C_{\text{in}}C_{\text{out}}$$

weights. A $1\times1$ reduction to $C_{\text{mid}}$ channels followed by that spatial convolution costs

$$C_{\text{in}}C_{\text{mid}}+K^2C_{\text{mid}}C_{\text{out}}.$$

When $C_{\text{mid}}\ll C_{\text{in}}$, the branch can inspect a larger spatial scale without paying the full dense channel cost. The caveat is that branch widths, reductions, and pooling choices are design decisions, not a theorem that any multi-branch block is automatically better.

Residual And Dense Routes

A residual block writes

$$x_{l+1}=x_l+F_l(x_l),$$

when the shapes match. The identity term gives the block a direct route: if $F_l$ initially behaves like a small correction, the network can represent a near-identity mapping and learn changes around it. For backpropagation, the local derivative contains an identity contribution:

$$\frac{\partial x_{l+1}}{\partial x_l}=I+\frac{\partial F_l}{\partial x_l}.$$

This does not make all optimization easy, but it explains why the shortcut is not merely a wiring flourish. It changes the paths through which activations and gradients can move.

This is the shape-preserving identity-shortcut case. Projection shortcuts, downsampling, post-add activations, and normalization change the local Jacobian story, so the identity derivative is a useful first model rather than the whole ResNet training theory.

DenseNet makes a different choice. Each layer consumes a concatenation of earlier feature maps:

$$x_l=H_l([x_0,x_1,\ldots,x_{l-1}]).$$

If every layer adds $k$ new channels and the initial block has $C_0$ channels, then layer $l$ sees roughly

$$C_0+k(l-1)$$

input channels. Counting layer $l$ as reading the initial state and all earlier layer outputs, the incoming dense links over $L$ layers sum to $1+2+\cdots+L$, so the dense directed connections among feature states grow like

$$\frac{L(L+1)}2.$$

That is the feature-reuse tradeoff: later layers can read earlier features directly, but concatenated state must be managed carefully.

ConvNeXt revisits this family with modern ConvNet blocks. A simplified depthwise $K\times K$ spatial convolution costs $K^2C$ weights because each channel has its own spatial filter, while a dense $K\times K$ convolution costs $K^2C^2$. Pointwise $1\times1$ layers then mix channels. Depthwise-separable arithmetic explains one component ConvNeXt adopts; it is not a claim that depthwise convolution is ConvNeXt-specific. Again, this is a simplified parameter story; the real architecture also depends on stage widths, downsampling, normalization, activation placement, training recipes, and data scale.