Vieta jumping: (a² + b²)/(ab + 1) is a square

Idea. Assume $k$ is not a perfect square. Among all pairs of positive integers $(a,b)$ with $a^2+b^2=k(ab+1)$ pick one with $a+b$ minimal, and with $a\ge b$. We build a smaller pair.

Case $a=b$. Then $2a^2=k(a^2+1)$, so $k=\frac{2a^2}{a^2+1}<2$. As $k$ is a positive integer, $k=1$, a perfect square. So assume $a>b$.

The jump. Regard the equation as $x^2-kb\,x+(b^2-k)=0$ in the unknown $x$. It has the root $x=a$; by Vieta the other root is
$$a'=kb-a=\frac{b^2-k}{a},$$
an integer.

• $a'>0$: if $a'<0$ then $a'\le-1$, and from $a'^2+kb|a'|+b^2-k=0$ (the equation with $x=a'$) together with $kb|a'|\ge k$ we get $a'^2+b^2\le0$, impossible. If $a'=0$ then $b^2=k$ would be a square, excluded. So $a'\ge1$.
• $a'<b$: indeed $a'=\dfrac{b^2-k}a<\dfrac{b^2}a\le b$ since $a\ge b$.

Thus $(a',b)$ is another pair of positive integers satisfying the equation, with $a'+b<a+b$. This contradicts the minimality of $a+b$.

Conclusion. $k$ must be a perfect square.

Check. $(a,b)=(8,2)$: $\frac{64+4}{17}=4$; $(27,3)$: $\frac{738}{82}=9$; $(30,8)$: $\frac{964}{241}=4$ ✓.

Remark. This is IMO 1988 Problem 6, the most famous Vieta jumping problem.

Lesson. Vieta jumping: view the equation as a quadratic in one variable, use the second root to produce a smaller solution, and conclude by minimality.