Lipschitz estimates for the norm and for inversion

Idea. The norm is Lipschitz by the triangle inequality; the inverse is Lipschitz thanks to an algebraic identity expressing $A^{-1}-B^{-1}$ through $B-A$.

Part 1: the norm.
- $\|A\|=\|(A-B)+B\|\le\|A-B\|+\|B\|$, so $\|A\|-\|B\|\le\|A-B\|$.
- Exchanging $A$ and $B$ gives $\|B\|-\|A\|\le\|B-A\|=\|A-B\|$.

Hence $\big|\|A\|-\|B\|\big|\le\|A-B\|$.

Part 2: inversion.
- *The identity.* For invertible $A,B$:
$$A^{-1}(B-A)B^{-1}=A^{-1}BB^{-1}-A^{-1}AB^{-1}=A^{-1}-B^{-1}.$$
- *The estimate.* By submultiplicativity,
$$\|A^{-1}-B^{-1}\|\le\|A^{-1}\|\,\|B-A\|\,\|B^{-1}\|\le M\cdot\|A-B\|\cdot M=M^2\|A-B\|.$$

Here both $A$ and $B$ must lie in the set $\{\|\,\cdot^{-1}\|\le M\}$.

Sharpness. For $A=\operatorname{diag}(\frac1M,1,\dots,1)$ and $B=\operatorname{diag}(\frac2M,1,\dots,1)$ we have $\|A^{-1}\|=M$, $\|B^{-1}\|=\frac M2\le M$, $\|A-B\|=\frac1M$ and $\|A^{-1}-B^{-1}\|=\left|M-\frac M2\right|=\frac M2$. The ratio is $\frac{M/2}{1/M}=\frac{M^2}2$: the constant $M^2$ cannot be improved beyond a factor $2$.

Remark. The set $\{\|A^{-1}\|\le M\}$ is exactly $\{\sigma_{\min}(A)\ge1/M\}$, the matrices at distance $\ge1/M$ from the singular matrices (Problem 63). So inversion is Lipschitz *away from the singular set*, with a constant blowing up like $\operatorname{dist}^{-2}$ and Problem 73 shows that it fails to be uniformly continuous globally.