Orthogonality, Projections, and Least-Squares Geometry

Let $U$ be a subspace of $\mathbb R^n$ spanned by columns

$$B = [b_1,\ldots,b_m] \in \mathbb R^{n\times m}.$$

Any point in $U$ can be written as $B\lambda$, where $\lambda\in\mathbb R^m$ is a coordinate vector. The projection of $x\in\mathbb R^n$ onto $U$ solves

$$\pi_U(x)=\arg\min_{z\in U}\|x-z\|_2.$$

Writing $z=B\lambda$, the residual is

$$r = x - B\lambda.$$

At the closest point, this residual is orthogonal to every spanning direction:

$$b_i^{\mathsf T}r = 0 \qquad i=1,\ldots,m.$$

Stacking those $m$ dot-product checks gives the normal equations for projection:

$$B^{\mathsf T}(x-B\lambda)=0.$$

Equivalently,

$$B^{\mathsf T}B\lambda = B^{\mathsf T}x.$$

If the columns of $B$ are independent, then $B^{\mathsf T}B$ is invertible and

$$\lambda = (B^{\mathsf T}B)^{-1}B^{\mathsf T}x.$$

The projected point is

$$\pi_U(x)=B(B^{\mathsf T}B)^{-1}B^{\mathsf T}x.$$

The matrix

$$P_U = B(B^{\mathsf T}B)^{-1}B^{\mathsf T}$$

is the projection matrix onto the column space of $B$ under the usual dot product.

Two caveats matter:

1. The projected point $\pi_U(x)$ is unique, but the coefficients $\lambda$ may not be unique when the columns of $B$ are dependent.
2. The inverse formula is a teaching formula. Numerically, explicit inverses can be fragile; solvers based on least squares, QR, SVD, pseudoinverses, or regularization are usually better.

Now let $X\in\mathbb R^{n\times d}$ be a design matrix and $y\in\mathbb R^n$ a target vector. Linear regression predicts

$$\hat y = Xw.$$

Least squares minimizes

$$\frac12\|y-Xw\|_2^2.$$

The zero-gradient condition first gives

$$X^{\mathsf T}(X\hat w-y)=0.$$

Multiplying by $-1$, the same condition can be written in the residual direction:

$$X^{\mathsf T}(y-X\hat w)=0.$$

or, using the residual $r=y-X\hat w$,

$$X^{\mathsf T}r=0.$$

That is the same projection condition: the fitted prediction $X\hat w$ is the closest point to $y$ in the column space of $X$, and the residual is orthogonal to every feature column.