Convergence of the matrix exponential and its first bounds

Idea. The series of norms is dominated by the scalar series $\sum\|A\|^k/k!$, which converges.

Absolute convergence. Submultiplicativity gives $\|A^k\|\le\|A\|^k$ by induction on $k$ (with $A^0=I$ and $\|I\|=1$ for the operator norm). Hence
$$\sum_{k\ge0}\left\|\frac{A^k}{k!}\right\|\le\sum_{k\ge0}\frac{\|A\|^k}{k!}=e^{\|A\|}<\infty.$$
Since $M_n(\mathbb{R})$ is complete (it is $\mathbb{R}^{n^2}$), an absolutely convergent series converges, so $e^A$ is well defined.

The two bounds. By the triangle inequality for series,
$$\|e^A\|\le\sum_{k\ge0}\frac{\|A\|^k}{k!}=e^{\|A\|},$$
and, removing the term $k=0$ (which is $I$),
$$\|e^A-I\|=\Big\|\sum_{k\ge1}\frac{A^k}{k!}\Big\|\le\sum_{k\ge1}\frac{\|A\|^k}{k!}=e^{\|A\|}-1.$$

Continuity. On the ball $\{\|A\|\le R\}$ the $k$-th term is bounded by $R^k/k!$, and $\sum R^k/k!<\infty$. By the Weierstrass M-test the series converges uniformly on this ball. Each partial sum $\sum_{k\le N}A^k/k!$ is a polynomial in the entries, hence continuous, and a uniform limit of continuous functions is continuous. So $\exp$ is continuous on every ball, hence on $M_n(\mathbb{R})$.

Remark. (For the Frobenius norm $\|I\|=\sqrt n$, and the first bound would need the factor $\sqrt n$; the second bound is unchanged.) The bound $\|e^A-I\|\le e^{\|A\|}-1$ shows that $e^A$ is close to $I$ when $A$ is small: this is the starting point of the local study of $\exp$ and $\log$ near $I$.