The operator-norm ball and O(2) in complex coordinates

Part 1. From the previous problem, $\|A\|=\sigma_1=|z|+|w|$. So the ball is $\{|z|+|w|\le1\}$.

Part 2. $A$ is orthogonal iff $A^TA=I$ iff both singular values equal $1$, i.e.
$$|z|+|w|=1\quad\text{and}\quad\bigl||z|-|w|\bigr|=1.$$
If $|z|\ge|w|$, the second equation is $|z|-|w|=1$, and subtracting from the first gives $|w|=0$, $|z|=1$. If $|w|>|z|$, then $|w|-|z|=1$ and we get $z=0$, $|w|=1$. So
$$O(2)=\{|z|=1,w=0\}\cup\{z=0,|w|=1\}.$$
The determinant is $|z|^2-|w|^2$, equal to $1$ on the first piece and $-1$ on the second. So the first piece is $SO(2)$ and the second is $O(2)\setminus SO(2)$ (rotations: $z=e^{i\theta}$, reflections: $w$ of modulus $1$).

Part 3: extreme points. The ball is $B=\{(z,w)\in\mathbb{C}^2:|z|+|w|\le1\}$.

*Points that are not extreme.*
- If $|z|+|w|<1$ and $(z,w)\ne0$, the point is the midpoint of the two points $(1\pm\eta)(z,w)$ of $B$ for small $\eta>0$; for $(z,w)=0$ it is the midpoint of $\pm(z_0,w_0)$ with $(z_0,w_0)\in B$ nonzero.
- If $|z|+|w|=1$ and $zw\ne0$, then
$$(z,w)=|z|\Bigl(\tfrac z{|z|},0\Bigr)+|w|\Bigl(0,\tfrac w{|w|}\Bigr),$$
a combination with weights $|z|,|w|\in(0,1)$ of weights summing to $1$ of two *distinct* points of $B$. So it is not extreme.

*Points of $O(2)$ are extreme.* Let $(z,0)$ with $|z|=1$, and suppose $(z,0)=\lambda(z_1,w_1)+(1-\lambda)(z_2,w_2)$ with $0<\lambda<1$ and $(z_i,w_i)\in B$. Then $w$-components give $\lambda w_1+(1-\lambda)w_2=0$ and
$$1=|\lambda z_1+(1-\lambda)z_2|\le\lambda|z_1|+(1-\lambda)|z_2|\le\lambda(1-|w_1|)+(1-\lambda)(1-|w_2|)\le1.$$
All inequalities are equalities, so $w_1=w_2=0$, $|z_1|=|z_2|=1$, and equality in the triangle inequality for unit complex numbers forces $z_1=z_2$ (the unit circle contains no segment). Hence the two points coincide and $(z,0)$ is extreme. The case $z=0$, $|w|=1$ is identical with the roles exchanged.

So the extreme points are exactly $\{|z|=1,w=0\}\cup\{z=0,|w|=1\}=O(2)$.

Remark. The ball is the convex hull of its extreme points (Krein-Milman), i.e. of two circles lying in two orthogonal planes of $\mathbb{R}^4$; this is the $n=2$ case of "the unit ball of the operator norm is the convex hull of $O(n)$". The ball is not strictly convex: it contains the segment joining $(1,0)$ and $(0,1)$.

Lesson. In explicit coordinates, extreme points can be found by splitting non-extreme points as convex combinations and testing the equality cases of the triangle inequality for the rest.