Matrix Decompositions: Eigendecomposition, SVD, and Spectral Structure

Let

$$A \in \mathbb R^{n\times n}.$$

A nonzero vector $p_i\in\mathbb R^n$ is an eigenvector of $A$ with eigenvalue $\lambda_i$ when

$$Ap_i = \lambda_i p_i.$$

If $A$ has $n$ linearly independent eigenvectors, stack them as columns of

$$P = [p_1,\ldots,p_n].$$

Then $P$ is invertible, and

$$A = P\Lambda P^{-1},$$

where

$$\Lambda = \operatorname{diag}(\lambda_1,\ldots,\lambda_n).$$

This is eigendecomposition. It says: change into the eigenvector basis, scale each coordinate by its eigenvalue, then change back.

The important caveat is that not every square matrix has enough independent eigenvectors. When it does not, this diagonalization story is not available in this simple form.

For a real symmetric matrix

$$S = S^{\mathsf T}\in\mathbb R^{n\times n},$$

the spectral theorem gives a stronger guarantee. There is an orthogonal matrix $Q$, whose columns are orthonormal eigenvectors, and a real diagonal matrix $\Lambda$ such that

$$S = Q\Lambda Q^{\mathsf T}.$$

Because $Q^{-1}=Q^{\mathsf T}$, the geometry is clean: orthonormal basis change, independent scaling, orthonormal basis change back.

Now let

$$A\in\mathbb R^{m\times n}.$$

The singular value decomposition is

$$A = U\Sigma V^{\mathsf T},$$

where $U\in\mathbb R^{m\times m}$ and $V\in\mathbb R^{n\times n}$ are orthogonal in the full SVD, and $\Sigma\in\mathbb R^{m\times n}$ is rectangular diagonal. Its diagonal entries are singular values

$$\sigma_1\ge \sigma_2\ge \cdots \ge \sigma_r>0,$$

where $r=\operatorname{rank}(A)$. In reduced form, we keep only the $r$ nonzero channels:

$$A = \sum_{i=1}^{r}\sigma_i u_i v_i^{\mathsf T}.$$

Each term is one rank-1 channel. The right singular vector $v_i$ is an input direction, the scalar $\sigma_i$ is the gain, and the left singular vector $u_i$ is the output direction:

$$Av_i = \sigma_i u_i.$$

The link back to eigenvalues is:

$$A^{\mathsf T}A v_i = \sigma_i^2 v_i,$$

and

$$AA^{\mathsf T}u_i = \sigma_i^2 u_i.$$

So right singular vectors are eigenvectors of $A^{\mathsf T}A$, left singular vectors are eigenvectors of $AA^{\mathsf T}$, and singular values are square roots of the corresponding nonzero eigenvalues.

The singular values themselves are fixed, but the displayed singular vectors are not always unique. Flipping the signs of both $u_i$ and $v_i$ leaves $\sigma_i u_i v_i^{\mathsf T}$ unchanged. If singular values are tied, any orthonormal basis inside that tied subspace is valid. The basis chosen for zero singular directions is also non-unique.

If $\sigma_i=0$, that channel vanishes. The zero right-singular directions span the null space. A tiny nonzero singular value means the channel exists in exact algebra but is numerically fragile.

The rank-$k$ truncated SVD keeps only the $k$ strongest channels:

$$A_k = \sum_{i=1}^{k}\sigma_i u_i v_i^{\mathsf T}.$$

Eckart-Young says this is the best rank-$k$ approximation to $A$ in spectral norm:

$$\|A-A_k\|_2 = \sigma_{k+1}.$$

This theorem is powerful, but it should be read with scope. It optimizes a matrix approximation norm. A real learning task may care about labels, fairness, robustness, interpretability, or downstream loss, so the largest singular values are not automatically the only values that matter.