Change of basis for a linear map

Why $P^{-1}AP$. Let $x$ have coordinates $X'$ in $\mathcal{B}$ and $X=PX'$ in the canonical basis. Then $f(x)$ has canonical coordinates $AX=APX'$, hence coordinates $P^{-1}APX'$ in $\mathcal{B}$.

Matrix $P$.
$$P = \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}, \qquad \det P=-2, \qquad P^{-1} = \frac12\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}.$$

New matrix.
$$AP=\begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} = \begin{pmatrix} 3 & 1 \\ 3 & -1 \end{pmatrix},$$
$$P^{-1}AP=\frac12\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}\begin{pmatrix} 3 & 1 \\ 3 & -1 \end{pmatrix} = \frac12\begin{pmatrix} 6 & 0 \\ 0 & 2 \end{pmatrix} = \begin{pmatrix} 3 & 0 \\ 0 & 1 \end{pmatrix}.$$

Conclusion. In $\mathcal{B}$ the matrix of $f$ is $\operatorname{diag}(3,1)$. This is expected: $f(1,1)=3(1,1)$ and $f(1,-1)=(1,-1)$, so the vectors of $\mathcal{B}$ are eigenvectors for the eigenvalues $3$ and $1$.