M_n(R)/O(n) is the cone of positive semidefinite matrices

Idea. A continuous closed surjection is a quotient map. So it suffices to show that $p$ is a closed continuous surjection whose fibres are the orbits.

Step 1: values and surjectivity. $A^TA$ is symmetric positive semidefinite ($x^TA^TAx=|Ax|^2\ge0$). Conversely if $S\in\mathrm{Sym}_n^+$, then $S=(S^{1/2})^TS^{1/2}=p(S^{1/2})$ with $S^{1/2}$ the positive square root. So $p:M_n\to\mathrm{Sym}_n^+$ is onto, and continuous (polynomial).

Step 2: constant on orbits. $(UA)^T(UA)=A^TU^TUA=A^TA$.

Step 3: fibres are orbits. Suppose $A^TA=B^TB$. Then $|Ax|^2=x^TA^TAx=|Bx|^2$ for all $x$. Define $V:\operatorname{im}A\to\operatorname{im}B$ by $V(Ax)=Bx$. It is well defined (if $Ax=Ax'$ then $|B(x-x')|=|A(x-x')|=0$) and norm-preserving, so it is an isometry between subspaces of equal dimension ($\operatorname{rank}A=\operatorname{rank}B$ since $\ker A=\ker B$). Extend $V$ by any isometry $(\operatorname{im}A)^\perp\to(\operatorname{im}B)^\perp$ to an orthogonal matrix $U$. Then $UAx=Bx$ for all $x$, i.e. $B=UA$.

Step 4: $p$ is closed. Let $F\subset M_n$ be closed and $p(A_k)\to S$ with $A_k\in F$. Then $\|A_k\|_F^2=\operatorname{tr}(A_k^TA_k)=\operatorname{tr}p(A_k)$ is bounded, so a subsequence $A_{k_j}\to A$; $A\in F$ and $p(A)=S$ by continuity. So $S\in p(F)$, and $p(F)$ is closed.

Step 5: conclusion. $p$ is a closed continuous surjection, hence a quotient map; since its fibres are the $O(n)$-orbits, $\mathrm{Sym}_n^+\cong M_n(\mathbb{R})/O(n)$.

Remark. The inverse can be seen as polar decomposition: the class of $A$ corresponds to the positive part $|A|=(A^TA)^{1/2}$, and $A=U|A|$ with $U$ orthogonal.

Lesson. To identify an orbit space, find a continuous closed (or proper) map that separates orbits.