The orthogonal group is closed and nowhere dense

Idea. $O(n)$ is cut out by polynomial equations, so it is closed. It is "thin" because it sits inside a sphere, and spheres are thin.

Step 1: closed. The map $\Phi:A\mapsto A^TA-I$ is polynomial in the entries, hence continuous, and $O(n)=\Phi^{-1}(\{0\})$.

Step 2: $O(n)\subset S$, a sphere. If $A^TA=I$ then $\|A\|_F^2=\operatorname{tr}(A^TA)=\operatorname{tr}I=n$. So $O(n)\subset S=\{\|A\|_F=\sqrt n\}$.

Step 3: a sphere has empty interior. Let $B\in S$ and $\varepsilon>0$. Then $(1+\varepsilon)B$ satisfies $\|(1+\varepsilon)B\|_F=(1+\varepsilon)\sqrt n>\sqrt n$, so $(1+\varepsilon)B\notin S$, while $\|(1+\varepsilon)B-B\|_F=\varepsilon\sqrt n\to0$. So no ball around $B$ lies in $S$.

Step 4: nowhere dense. A subset of a set with empty interior has empty interior, and $O(n)$ is closed, so $\overline{O(n)}=O(n)$ has empty interior: $O(n)$ is nowhere dense.

Remark.
- $O(n)$ is in fact compact (closed and bounded) and is a manifold of dimension $\frac{n(n-1)}2$, much smaller than $n^2$.
- A sharper statement is that the orthogonal group is the set of extreme points of the operator-norm unit ball (Part 1).