Ky Fan's maximum principle

Idea. Reduce to a diagonal $S$. Then $\operatorname{tr}(\Lambda P')=\sum\lambda_ip'_{ii}$ is a weighted sum with weights in $[0,1]$ adding up to $k$, which is maximal when the weights sit on the $k$ largest $\lambda_i$.

Step 1: reduction. Let $S=Q\Lambda Q^T$ and $P$ an orthogonal projection of rank $k$ (i.e. $P=P^T=P^2$, $\operatorname{tr}P=k$). Put $P'=Q^TPQ$; it is again an orthogonal projection of rank $k$ and
$$\operatorname{tr}(SP)=\operatorname{tr}(Q\Lambda Q^TP)=\operatorname{tr}(\Lambda Q^TPQ)=\operatorname{tr}(\Lambda P')=\sum_i\lambda_ip'_{ii}.$$

Step 2: the weights. Let $p_i=p'_{ii}$. Then $p_i=e_i^TP'e_i=e_i^TP'^TP'e_i=|P'e_i|^2\in[0,1]$ (because $P'$ is a projection, $|P'v|\le|v|$), and $\sum_ip_i=\operatorname{tr}P'=k$.

Step 3: the bound. Write
$$\sum_i\lambda_ip_i-k\lambda_k=\sum_i(\lambda_i-\lambda_k)p_i.$$
For $i\le k$, $\lambda_i-\lambda_k\ge0$ and $p_i\le1$, so those terms are at most $\lambda_i-\lambda_k$. For $i>k$, $\lambda_i-\lambda_k\le0$ and $p_i\ge0$, so those terms are $\le0$. Therefore
$$\sum_i\lambda_ip_i-k\lambda_k\le\sum_{i\le k}(\lambda_i-\lambda_k)=\sum_{i\le k}\lambda_i-k\lambda_k,$$
that is $\operatorname{tr}(SP)\le\lambda_1+\dots+\lambda_k$.

Step 4: attained. Let $P_0$ be the orthogonal projection onto the span of the first $k$ columns of $Q$ (the top $k$ eigenvectors). Then $\operatorname{tr}(SP_0)=\lambda_1+\dots+\lambda_k$. So the maximum exists (also because the set of such projections is compact) and equals the sum.

Step 5: convexity. For each fixed $P$, $S\mapsto\operatorname{tr}(SP)$ is linear. The function $S\mapsto\lambda_1+\dots+\lambda_k$ is the maximum of these linear functions, hence convex.

Remark. For $k=1$ this is the Rayleigh formula for $\lambda_{\max}$ (previous problem). For $k=n$ it is the trace, which is linear.

Lesson. Variational formulas as maxima of linear functions give convexity for free.