Training Loss Curves and Debugging

Let $\theta_t$ be the model parameters after training step $t$. Let $\ell(f_{\theta}(x), y)$ be the loss for one labeled example. For a minibatch $\mathcal B_t$ and validation set $\mathcal V$, define

$$\hat L_{\mathrm{train}}(t) = \frac{1}{|\mathcal B_t|} \sum_{(x_i,y_i)\in\mathcal B_t} \ell(f_{\theta_t}(x_i), y_i),$$

and

$$\hat L_{\mathrm{val}}(t) = \frac{1}{|\mathcal V|} \sum_{(x_i,y_i)\in\mathcal V} \ell(f_{\theta_t}(x_i), y_i),$$

where validation is measured without training updates and with validation-mode layer behavior.

The simplest overfitting signal is the same-metric generalization gap

$$G_t = \hat L_{\mathrm{val}}(t) - \hat L_{\mathrm{train}}(t).$$

A large positive $G_t$ late in training says the model is fitting the training stream better than the validation stream. It does not say why. The cause may be model capacity, regularization, data scarcity, distribution shift, leakage in the opposite direction, validation noise, or a measurement bug.

A local improvement score over a window of $k$ steps is

$$\Delta^{(k)}_{\mathrm{train}}(t) = \hat L_{\mathrm{train}}(t-k) - \hat L_{\mathrm{train}}(t).$$

If both $\Delta^{(k)}_{\mathrm{train}}(t)$ and $\Delta^{(k)}_{\mathrm{val}}(t)$ are small while both losses remain high, suspect underfitting, a too-small learning rate, a label/objective mismatch, frozen parameters, or a broken training loop.

A crude noise score is the standard deviation of recent first differences:

$$N^{(k)}_{\mathrm{train}}(t) = \mathrm{Std}\left( \hat L_{\mathrm{train}}(j) - \hat L_{\mathrm{train}}(j-1) \right)_{j=t-k+1}^{t}.$$

High $N$ can be healthy minibatch noise, but if it comes with upward spikes, divergence, or non-finite values, check the learning rate, gradient scale, numerical stability, batch construction, and loss implementation.

For checkpoint debugging, define the run contract at step $t$ as

$$C_t = (\theta_t,\ s_t,\ q_t,\ a_t,\ r_t,\ t,\ c),$$

where $s_t$ is optimizer state, $q_t$ scheduler state, $a_t$ mixed-precision scaler state when used, $r_t$ RNG/data-order state when exact continuation matters, and $c$ the run config. If a curve improves, resumes, and then repeats or jumps, the question is whether the actual resume restored the same $C_t$.