Train/Dev/Test Splits, Cross-Validation, and Leakage

Let the full dataset be

$$D=\{(x_i,y_i)\}_{i=1}^n.$$

A train/dev/test split partitions the row indices into three disjoint sets whose union is the dataset:

$$I_{\text{train}}\cup I_{\text{dev}}\cup I_{\text{test}}=\{1,\ldots,n\},\quad I_{\text{train}}\cap I_{\text{dev}}=\varnothing,\quad I_{\text{train}}\cap I_{\text{test}}=\varnothing,\quad I_{\text{dev}}\cap I_{\text{test}}=\varnothing.$$

For a hyperparameter value $\lambda$, the training algorithm produces a fitted model using only the training rows:

$$\hat f_{\lambda}=A_{\lambda}(D_{I_{\text{train}}}).$$

The development loss estimates which candidate looks best:

$$\widehat R_{\text{dev}}(\lambda)= \frac{1}{|I_{\text{dev}}|} \sum_{i\in I_{\text{dev}}} L(\hat f_{\lambda}(x_i),y_i).$$

Model selection is the act of choosing

$$\hat\lambda=\arg\min_{\lambda\in\Lambda}\widehat R_{\text{dev}}(\lambda).$$

After that choice is frozen, the test loss is computed once:

$$\widehat R_{\text{test}}= \frac{1}{|I_{\text{test}}|} \sum_{i\in I_{\text{test}}} L(\hat f_{\hat\lambda}(x_i),y_i).$$

This number estimates the performance of the selected procedure only if $I_{\text{test}}$ did not influence $A_\lambda$, the preprocessing, the feature selection, the prompt, the stopping rule, the threshold, or $\hat\lambda$.

Some protocols refit the selected pipeline on $I_{\text{train}}\cup I_{\text{dev}}$ before this one test evaluation. That is valid only if the refit rule was fixed before reading the test set. The invariant is not "never use dev again"; it is "never let test influence the selected procedure."

With $K$-fold cross-validation, we partition the training budget $I_{\text{train}}$ into folds $F_1,\ldots,F_K$. For each $\lambda$, fit on all folds except $F_k$ and validate on $F_k$. A per-example version is

$$\widehat R_{\text{cv}}(\lambda)= \frac{1}{|I_{\text{train}}|}\sum_{k=1}^K \sum_{i\in F_k} L\!\left(A_\lambda(D_{I_{\text{train}}\setminus F_k})(x_i),y_i\right).$$

When all folds have the same size, this is the same as averaging the fold losses.

Then choose $\hat\lambda$ by cross-validation, refit the chosen pipeline on the full training budget, and evaluate on the untouched test set.

Leakage appears when the fitted pipeline depends on held-out rows. Suppose $T$ is a preprocessing operation such as scaling, imputation, PCA, or feature selection. A clean pipeline is

$$\hat T_{\text{train}}=B(D_{I_{\text{train}}}),\qquad \hat f=A(\hat T_{\text{train}}(D_{I_{\text{train}}})).$$

The same learned transform $\hat T_{\text{train}}$ may be applied to dev/test inputs, but it must not be fitted from dev/test rows. The leaky version is

$$\hat T_{\text{all}}=B(D_{I_{\text{train}}}\cup D_{I_{\text{dev}}}\cup D_{I_{\text{test}}}),$$

then evaluating on $I_{\text{test}}$ after the test rows helped define the feature space. That reported score answers the wrong question: "how well does a pipeline do after seeing statistics from the evaluation set?"

Inside cross-validation, "training rows" means $I_{\text{train}}\setminus F_k$ for the current fold. A scaler, imputer, PCA map, or feature selector fitted once on all of $I_{\text{train}}$ before scoring folds has already leaked validation-fold information into the CV estimate.

For time series, groups, people, papers, or benchmark items with near-duplicates, the disjoint-index condition is not enough. The split must respect the unit of generalization: future time after past time, new users after old users, new papers after training papers, or new problem families after seen families.