The distance from a 2×2 matrix to the rotation group

Idea. In the coordinates $(z,w)$ the Frobenius norm is Euclidean (up to the factor $\sqrt2$), so the nearest rotation to $A$ is the nearest point of the circle $\{(e^{i\alpha},0)\}$ to the point $(z,w)$.

Step 1: the squared distance. $A-R_\alpha$ has coordinates $(z-e^{i\alpha},\,w)$ since $A\mapsto(z,w)$ is linear. Hence
$$\|A-R_\alpha\|_F^2=2\bigl(|z-e^{i\alpha}|^2+|w|^2\bigr).$$

Step 2: minimise over $\alpha$. For any $\alpha$, the reverse triangle inequality gives $|z-e^{i\alpha}|\ge\bigl||z|-|e^{i\alpha}|\bigr|=\bigl||z|-1\bigr|$. Equality holds iff $e^{i\alpha}$ and $z$ are positively proportional, i.e. $e^{i\alpha}=z/|z|$ when $z\ne0$. When $z=0$ we have $|0-e^{i\alpha}|=1=|0-1|$ for all $\alpha$, so the bound is attained for every $\alpha$. Therefore
$$\min_\alpha\|A-R_\alpha\|_F^2=2\Bigl((|z|-1)^2+|w|^2\Bigr).$$

Nearest rotation. If $z\ne0$ the minimiser is unique: $R=R_\alpha$ with $e^{i\alpha}=z/|z|$, i.e. the rotation by $\arg z$ (in matrices: $R=\frac1{|z|}\begin{pmatrix}\operatorname{Re}z&-\operatorname{Im}z\\\operatorname{Im}z&\operatorname{Re}z\end{pmatrix}$, the normalised "rotation-scaling part" $\frac{(a+d)}{2}I+\frac{(c-b)}2\begin{pmatrix}0&-1\\1&0\end{pmatrix}$ of $A$). If $z=0$ every rotation is equally near (this happens iff $A$ is "anti-conformal", of the form $x\mapsto w\bar x$).

Check. $A=2I$: $z=2$, $w=0$; the formula gives $2(2-1)^2=2$, and indeed $\|2I-I\|_F^2=2$. $A=\operatorname{diag}(1,-1)$: $z=0$, $w=1$; the formula gives $2(1+1)=4$, and $\|\operatorname{diag}(1,-1)-R_\alpha\|_F^2=(1-\cos\alpha)^2+\sin^2\alpha+\sin^2\alpha+(-1-\cos\alpha)^2=4$ for every $\alpha$. A numerical minimisation over a fine grid of $\alpha$ on random matrices reproduces the formula.

Remark. For the other component of $O(2)$, $\{z=0,|w|=1\}$, the same argument gives $\min\|A-F\|_F^2=2\bigl(|z|^2+(|w|-1)^2\bigr)$. Comparing the two minima, $\bigl[(|z|-1)^2+|w|^2\bigr]-\bigl[|z|^2+(|w|-1)^2\bigr]=2(|w|-|z|)$, so a matrix is closer to $SO(2)$ than to $O(2)\setminus SO(2)$ iff $|z|>|w|$, i.e. iff $\det A>0$. This is the $2\times2$ case of the orthogonal Procrustes problem: the nearest orthogonal matrix is obtained from the SVD $A=U\Sigma V^T$ as $UV^T$.

Lesson. When a distance to a group is wanted, pick coordinates in which the group is a coordinate subset (here the circle in the $z$-plane) and the norm is Euclidean.