The set {ρ(A) < 1}: open, with boundary {ρ = 1}

Idea. Once $\rho$ is known to be continuous, the topology follows from the homogeneity $\rho(tA)=|t|\rho(A)$, exactly as for a norm (even though $\rho$ is not a norm).

Step 1: $\rho$ is continuous. Let $A_k\to A$. The characteristic polynomials $\chi_{A_k}$ have coefficients converging to those of $\chi_A$. By the continuity of roots (Part 3: every root of $\chi_{A_k}$ is within $C\eta_k^{1/n}$ of a root of $\chi_A$ and vice versa, with $\eta_k\to0$), we get $|\rho(A_k)-\rho(A)|\to0$. (The estimate behind this: if $p,q$ are monic of degree $n$ with $|\mu|\le R$ a root of $q$ and coefficients differing by at most $\eta$, then $\prod_i|\mu-\lambda_i|=|p(\mu)|=|p(\mu)-q(\mu)|\le\eta\,n(1+R)^{n-1}$, so some root $\lambda_i$ of $p$ is within $(\eta n(1+R)^{n-1})^{1/n}$ of $\mu$; the coefficients of $\chi_{A_k}$ are bounded since $A_k$ is, and so are the roots.) Concretely: if $\mu$ is a root of $\chi_{A_k}$ of maximal modulus, some root $\lambda$ of $\chi_A$ has $|\mu-\lambda|\le C\eta_k^{1/n}$, so $\rho(A_k)\le\rho(A)+C\eta_k^{1/n}$; exchanging the roles gives the reverse inequality.

Step 2: open and closed sets. $\{\rho<1\}=\rho^{-1}((-\infty,1))$ is open and $\{\rho\le1\}=\rho^{-1}((-\infty,1])$ is closed.

Step 3: closure. Since $\{\rho<1\}\subset\{\rho\le1\}$ and the latter is closed, $\overline{\{\rho<1\}}\subset\{\rho\le1\}$. Conversely if $\rho(A)\le1$ then for $t\in(0,1)$: $\rho(tA)=t\rho(A)<1$ (eigenvalues of $tA$ are $t$ times those of $A$; if $\rho(A)=0$ then $\rho(tA)=0<1$), and $\|tA-A\|=(1-t)\|A\|\to0$. So $A\in\overline{\{\rho<1\}}$.

Step 4: boundary. The set $\{\rho<1\}$ is open, so $\partial\{\rho<1\}=\overline{\{\rho<1\}}\setminus\{\rho<1\}=\{\rho=1\}$.

Example. $A=\begin{pmatrix}1&5\\0&1\end{pmatrix}$ has $\rho=1$ (boundary) although $\|A\|\approx5.19>1$; $tA$ has $\rho=t<1$.

Remark.
- $\rho(A)<1$ iff $A^k\to0$ iff some norm satisfies $\|A\|<1$; but no single norm works for all of $\{\rho<1\}$: the set is not bounded and not convex. For the non-convexity take $N=\begin{pmatrix}0&2\\0&0\end{pmatrix}$ and $N^T$: $\rho(N)=\rho(N^T)=0$ but $\frac12(N+N^T)=\begin{pmatrix}0&1\\1&0\end{pmatrix}$ has $\rho=1$.
- The interior of $\{\rho\le1\}$ is $\{\rho<1\}$ (if $\rho(A)=1$ then $\rho(tA)=t>1$ for $t>1$ arbitrarily close).