P(x)P(x − 1) = P(x²)

Answer. $P=0$ and $P(x)=(x^2+x+1)^n$ for $n\ge0$.

These work. $(x^2+x+1)^n\bigl((x-1)^2+(x-1)+1\bigr)^n=(x^2+x+1)^n(x^2-x+1)^n=(x^4+x^2+1)^n=P(x^2)$.

Converse. Let $P\ne0$ be a solution and let $R$ be its (finite) set of complex roots. If $R=\varnothing$, $P$ is a constant $c$ with $c^2=c$, so $c=1$. Assume $R\ne\varnothing$ and let $a\in R$.

*Two closure properties.* Putting $x=a$ gives $P(a^2)=P(a)P(a-1)=0$, so $a^2\in R$. Putting $x=a+1$ gives $P((a+1)^2)=P(a+1)P(a)=0$, so $(a+1)^2\in R$.

*Roots have modulus $0$ or $1$.* By the first property, $a,a^2,a^4,\dots$ all lie in the finite set $R$, so $a^{2^j}=a^{2^k}$ for some $j<k$. Taking moduli, $|a|^{2^j}=|a|^{2^k}$, so $|a|\in\{0,1\}$. The same applies to the root $(a+1)^2$: either $(a+1)^2=0$ or $|a+1|=1$.

*$0$ and $-1$ are not roots.* If $0\in R$, then $(0+1)^2=1\in R$ and then $(1+1)^2=4\in R$, but $|4|\notin\{0,1\}$: contradiction. If $-1\in R$, then $(-1+1)^2=0\in R$, which we just excluded.

*All roots are $\omega,\bar\omega$.* So every root has $|a|=1$ and $|a+1|=1$. From $|a+1|^2=|a|^2+2\operatorname{Re}a+1=2+2\operatorname{Re}a=1$ we get $\operatorname{Re}a=-\tfrac12$, so $a=-\tfrac12\pm\tfrac{\sqrt3}2i$, the roots of $x^2+x+1$.

*Conclusion.* $P(x)=c\,(x-\omega)^p(x-\bar\omega)^q$. Real coefficients force $p=q=n$, so $P=c(x^2+x+1)^n$. Comparing leading coefficients gives $c^2=c$, so $c=1$. $\blacksquare$

Lesson. For $P(f(x))$-type equations, study how the equation moves roots around: it produces infinite orbits of roots, which a polynomial cannot have unless the orbit is finite.