PCA as Optimization and Eigenspace Projection

Let

$$X\in\mathbb R^{n\times d}$$

be a data matrix with $n$ observations as rows and $d$ features as columns. Define the feature mean

$$\mu = \frac1n X^{\mathsf T}\mathbf 1 \in\mathbb R^d,$$

and center the data:

$$X_c = X - \mathbf 1\mu^{\mathsf T}.$$

The sample covariance matrix is

$$S = \frac{1}{n-1}X_c^{\mathsf T}X_c \in\mathbb R^{d\times d}.$$

Some sources use $1/n$ instead of $1/(n-1)$. That changes the eigenvalues by a constant factor, but it does not change the eigenvectors or explained-variance ratios.

The first principal direction solves

$$v_1 = \arg\max_{\|v\|_2=1} v^{\mathsf T}Sv.$$

The objective is the sample variance of the one-dimensional scores

$$z_1 = X_cv_1.$$

Using a Lagrange multiplier for the constraint $\|v\|_2=1$ gives

$$Sv = \lambda v.$$

So the maximizer is an eigenvector of $S$. The best first direction is the eigenvector with the largest eigenvalue.

For $k$ components, collect the top orthonormal eigenvectors as columns of

$$V_k = [v_1,\ldots,v_k]\in\mathbb R^{d\times k}, \qquad V_k^{\mathsf T}V_k=I_k.$$

The loading directions $v_i$ are orthonormal. The score columns

$$Z_k = X_cV_k\in\mathbb R^{n\times k}$$

are uncorrelated because

$$\frac{1}{n-1}Z_k^{\mathsf T}Z_k = V_k^{\mathsf T}SV_k = \operatorname{diag}(\lambda_1,\ldots,\lambda_k).$$

The centered rank-$k$ reconstruction is

$$\widehat X_{c,k} = Z_kV_k^{\mathsf T} = X_cV_kV_k^{\mathsf T}.$$

Then add the mean back:

$$\widehat X_k = \widehat X_{c,k} + \mathbf 1\mu^{\mathsf T}.$$

The retained variance is

$$\operatorname{tr}(V_k^{\mathsf T}SV_k) = \sum_{i=1}^{k}\lambda_i.$$

The lost squared reconstruction energy is

$$\|X_c - X_cV_kV_k^{\mathsf T}\|_F^2.$$

PCA chooses the same $V_k$ for both objectives: it maximizes retained variance and minimizes this squared reconstruction error among rank-$k$ orthogonal projections.

The SVD makes the computation and reconstruction formula explicit. If

$$X_c = U\Sigma V^{\mathsf T},$$

then the principal directions are the right singular vectors in $V$. The rank-$k$ centered reconstruction is

$$\widehat X_{c,k} = U_k\Sigma_kV_k^{\mathsf T},$$

and the exact Frobenius reconstruction loss is

$$\|X_c-\widehat X_{c,k}\|_F^2 = \sum_{i>k}\sigma_i^2.$$

The explained variance for component $i$ is

$$\lambda_i = \frac{\sigma_i^2}{n-1},$$

and the explained-variance ratio is

$$\frac{\sigma_i^2}{\sum_j \sigma_j^2}.$$

Three caveats keep the geometry honest.

First, signs are arbitrary. If $v_i$ is a principal direction, then $-v_i$ describes the same undirected line. Second, tied eigenvalues make the basis inside the tied subspace non-unique. Third, PCA is sensitive to scaling and outliers. Standardizing features can be essential when units differ, but it is not automatic truth; if scale itself carries meaning, standardizing may erase useful structure.

When PCA is used before a supervised model, it must also obey the split discipline: fit centering, scaling, and PCA on the training split only, then transform development and test data with that fitted preprocessing.