The Frobenius coin problem for two coins

Normalising. Since $\gcd(a,b)=1$, $a$ is invertible modulo $b$, so for each integer $n$ there is a unique $x_0\in\{0,\dots,b-1\}$ with $ax_0\equiv n\pmod b$. Put $y_0=\frac{n-ax_0}b\in\mathbb Z$. Every integer solution of $ax+by=n$ is $(x_0+tb,\ y_0-ta)$ for $t\in\mathbb Z$.

Criterion. $n$ is representable with $x,y\ge0$ iff $y_0\ge0$. Indeed, a representation needs $x_0+tb\ge0$, which forces $t\ge0$, and then $y_0-ta\le y_0$ must be $\ge0$, so $y_0\ge0$. Conversely $y_0\ge0$ gives the representation $(x_0,y_0)$.

The number $n=ab-a-b$. We need $ax_0\equiv-a-b\equiv-a\pmod b$, so $x_0\equiv-1$, i.e. $x_0=b-1$. Then $y_0=\frac{ab-a-b-a(b-1)}b=\frac{-b}b=-1<0$. Not representable.

Larger $n$. If $n>ab-a-b$, then since $x_0\le b-1$,
$$y_0=\frac{n-ax_0}b\ge\frac{n-a(b-1)}b>\frac{ab-a-b-ab+a}b=-1,$$
so $y_0>-1$, and being an integer, $y_0\ge0$.

Example. For $a=5$, $b=7$ the largest amount that cannot be paid is $35-12=23$.

Lesson. Reduce one variable modulo the other coefficient to get a canonical representative, then the sign of the other variable decides everything.