Linear Regression & Least Squares

For $n$ training examples with $d$ features, collect the inputs in a design matrix $X\in\mathbb R^{n\times d}$ and targets in $y\in\mathbb R^n$. A linear model uses coefficients $w\in\mathbb R^d$ and predicts

$$\hat y = Xw.$$

The residual vector is

$$r = y - Xw.$$

Least squares minimizes the residual sum of squares:

$$J(w)=\frac12\lVert y-Xw\rVert_2^2.$$

The factor $\frac12$ only cleans up the derivative. If you instead average over examples, the gradient below is scaled by $\frac1n$; the minimizer is the same, but the step size in gradient descent changes. Expanding the unaveraged gradient gives

$$\nabla_w J(w)=X^{\mathsf T}(Xw-y).$$

At an interior optimum, the gradient is zero:

$$X^{\mathsf T}(Xw-y)=0.$$

So the fitted residual vector is orthogonal to every column of $X$:

$$X^{\mathsf T}r = 0.$$

That is the geometric heart of least squares. The prediction $X\hat w$ is the closest point to $y$ inside the column space of $X$. The residual is orthogonal to the feature columns, not magically orthogonal to the target vector, every data point in a scatterplot, or every visual segment you draw.

When $X^{\mathsf T}X$ is invertible, the normal equation has the closed-form solution

$$\hat w=(X^{\mathsf T}X)^{-1}X^{\mathsf T}y.$$

If $X^{\mathsf T}X$ is singular or badly conditioned, the formula may not exist or may be numerically fragile. Rank-deficient systems can have many minimizers; the pseudoinverse gives the minimum-norm solution. Practical code usually uses a least-squares solver, QR decomposition, SVD, or regularization rather than explicitly forming the inverse.

For a model with an intercept in one dimension, use columns $[1,x_i]$, so $w=[b,m]^{\mathsf T}$ and $\hat y_i=b+mx_i$.