The extreme points of the operator-norm unit ball

Idea. An orthogonal matrix sends every unit vector to a *unit* vector, i.e. to an extreme point of the Euclidean ball, which leaves no room for a non-trivial midpoint decomposition. A matrix with a singular value below $1$ has slack in that direction.

Step 1: orthogonal matrices are extreme. Let $U\in O(n)$ and $U=\frac{A+B}2$ with $\|A\|,\|B\|\le1$. For a unit vector $x$,
$$1=|Ux|=\Big|\frac{Ax+Bx}2\Big|\le\frac{|Ax|+|Bx|}2\le1.$$
Hence $|Ax|=|Bx|=1$. By the parallelogram law,
$$\Big|\frac{Ax+Bx}2\Big|^2=\frac{|Ax|^2+|Bx|^2}2-\Big|\frac{Ax-Bx}2\Big|^2=1-\frac{|Ax-Bx|^2}4,$$
and the left side is $|Ux|^2=1$, so $Ax=Bx$. This holds for all unit $x$, hence $A=B$ (and $=U$).

Step 2: other points of $\mathcal B$ are not extreme. Let $A\in\mathcal B$ be non-orthogonal, with SVD $A=P\Sigma Q^T$, $\Sigma=\operatorname{diag}(\sigma_i)$, $0\le\sigma_i\le1$. If all $\sigma_i=1$ then $A=PQ^T$ is orthogonal; so some $\sigma_i<1$. Choose real numbers $s\ne t$ in $[-1,1]$ with $\frac{s+t}2=\sigma_i$:
- if $0<\sigma_i<1$, take $\delta=\min(\sigma_i,1-\sigma_i)>0$ and $s=\sigma_i+\delta$, $t=\sigma_i-\delta$ (then $0\le t<s\le1$);
- if $\sigma_i=0$, take $s=1$, $t=-1$.

Let $\Sigma_s$, $\Sigma_t$ be $\Sigma$ with $\sigma_i$ replaced by $s$, resp. $t$. Both diagonal matrices have entries of absolute value $\le1$, so $A_s=P\Sigma_sQ^T$ and $A_t=P\Sigma_tQ^T$ belong to $\mathcal B$. They are distinct, and $\frac{A_s+A_t}2=A$. So $A$ is not extreme.

Remark. Compare with the Frobenius ball, which is strictly convex: *all* of its boundary points are extreme. The operator-norm ball has far fewer extreme points ($O(n)$, a manifold of dimension $n(n-1)/2$, inside a boundary of dimension $n^2-1$). The next problem shows these extreme points generate the whole ball by convex combinations (a Krein-Milman-type statement).