Maths WorkshopMaths problems with full solutions
Exercise 1478Calculus › Multivariable calculus › Optimisation in several variablesL3Added 2 Nov 2026

Least squares as an optimisation problem

#least squares#optimisation#regression

Statement

Let (x1,y1),…,(xn,yn)(x_1,y_1),\ldots,(x_n,y_n) be points. Minimize
F(m,p)=∑k=1n(yk−mxk−p)2F(m,p) = \sum_{k=1}^n (y_k - m x_k - p)^2
over R2\mathbb{R}^2.

Hint

Compute the partial derivatives ∂F∂m\frac{\partial F}{\partial m} and ∂F∂p\frac{\partial F}{\partial p}, set them to zero, and solve the system.

Show solutionHide solution

Solution

Partial derivatives.
∂F∂m=−2∑k=1nxk(yk−mxk−p)=0,\frac{\partial F}{\partial m} = -2 \sum_{k=1}^n x_k(y_k - m x_k - p) = 0,
∂F∂p=−2∑k=1n(yk−mxk−p)=0.\frac{\partial F}{\partial p} = -2 \sum_{k=1}^n (y_k - m x_k - p) = 0.

Normal system. Dividing by −2-2:
∑xkyk=m∑xk2+p∑xk,∑yk=m∑xk+np.\sum x_k y_k = m \sum x_k^2 + p \sum x_k, \qquad \sum y_k = m \sum x_k + n p.
From the second equation, p=yˉ−mxˉp = \bar y - m \bar x. Substituting into the first:
∑xkyk=m∑xk2+(yˉ−mxˉ)nxˉ=m(∑xk2−nxˉ2)+nxˉyˉ.\sum x_k y_k = m \sum x_k^2 + (\bar y - m \bar x) n \bar x = m \left( \sum x_k^2 - n \bar x^2 \right) + n \bar x \bar y.
Hence
m=∑xkyk−nxˉyˉ∑xk2−nxˉ2=∑(xk−xˉ)(yk−yˉ)∑(xk−xˉ)2.m = \frac{\sum x_k y_k - n \bar x \bar y}{\sum x_k^2 - n \bar x^2} = \frac{\sum (x_k - \bar x)(y_k - \bar y)}{\sum (x_k - \bar x)^2}.
This is the slope of the least-squares line.

Global minimum. FF is a convex quadratic function (sum of squares of affine forms), so the critical point is a global minimum.