Exercise 2169Linear algebra › Inner product spacesL1Added 3 Nov 2026

Least squares and normal equations

#least squares#normal equations#orthogonal projection

Statement

Let $A$ be an $m\times n$ matrix with full column rank, $b\in\mathbb{R}^m$ . Show that the vector $x^*$ minimizing $\|Ax-b\|$ satisfies the normal equations
$A^TAx^*=A^Tb.$

Hint

Interpret $Ax$ as a point in $\operatorname{Col}(A)$ , and $x^*$ as the orthogonal projection of $b$ .

Show solutionHide solution

Solution

Geometric interpretation. Minimizing $\|Ax-b\|$ over $x$ is equivalent to finding the point of $\operatorname{Col}(A)$ closest to $b$ . This is the orthogonal projection of $b$ onto $\operatorname{Col}(A)$ .

Orthogonality. For $x^*$ to be the projection, $b-Ax^*$ must be orthogonal to $\operatorname{Col}(A)$ , i.e. to every column of $A$ :
$A^T(b-Ax^*)=0.$

Normal equations. Rearranging:
$A^TAx^*=A^Tb.$
Since $A$ has full column rank, $A^TA$ is invertible, so $x^*=(A^TA)^{-1}A^Tb$ .

Conclusion. The least-squares solution satisfies the normal equations.

Least squares and normal equations

Statement

Solution

More problems in Inner product spaces