Distinguished cuspidal representations over \(p\)-adic and finite fields (Q2057572)

Let \(F\) be either a \(p\)-adic field or a finite field. Let \(G\) be a reductive group defined over \(F\) and let \(M\) be some subgroup of \(G(F)\). The distinction problem concerns the vanishing of the space \[ \mathrm{Hom}_{M}(\pi, 1) \] for an irreducible smooth representation \(\pi\) of \(G(F)\). When the above \(\mathrm{Hom}\) space is non-zero, the representation \(\pi\) is called \(M\)-distinguished. More generally, the authors are interested in the dimension of this \(\mathrm{Hom}\) space. A particular interesting case is when \(\pi\) is cuspidal and \(M\) is the group of \(F\)-points of the group fixed by an involution of \(G\) which is the case treated in this article. In the finite field case, an important class of cuspidal representations of \(G(F)\) are those obtained from Deligne-Lusztig induction of characters \(\rho\) in general position of \(L\), the \(F\)-points of a maximal anisotropic torus. Let \(\pi(\rho)\) denote such a cuspidal representation. In the \(p\)-adic field case, an important class of cuspidal representations are constructed by \textit{J.-K. Yu} [J. Am. Math. Soc. 14, No. 3, 579--622 (2001; Zbl 0971.22012)] which has been simplified by the author in a previous work. Let \(H\) be a \(F\)-subgroup of \(G\) such that \(Z_H/Z_G\) is anisotropic and \(H\) becomes a Levi subgroup after a tamely ramified base change. Let \(x\) be a point in the reduced building of \(H\). Let \(\rho\) be a permissible representation of \(L=H_x\), the stabilizer of \(x\) in \(H(F)\). The theory gives a cuspidal representation \(\pi(\rho)\) of \(G(F)\). Let \(\Theta\) denote a \(G(F)\)-orbit of an involution \(\theta\) of G, and let \[ \langle\Theta, \rho \rangle_G= \dim \mathrm{Hom}_{G^{\theta}(F)}(\pi(\rho), 1), \] which is independent of \(\theta\in \Theta\), where \(G^\theta\) is the group of fixed points of \(\theta\). The main result of the article is the formula: \[ \langle\Theta, \rho \rangle_G=\sum_{\vartheta \sim \rho} m_L(\vartheta)\langle\vartheta, \rho\rangle_L, \] where \(m_L(\vartheta)\) is an positive integer determined explicitly by \(\vartheta\), and the sum is taken over \(L\)-orbits \(\vartheta\) in \(\Theta\) such that \(\theta(L)=L\) and \(\langle\vartheta, \rho\rangle_L:=\dim \mathrm{Hom}_{L^\theta}(\rho, \varepsilon_{L,\theta})\) is non-zero for some \(\theta\in\vartheta\). Here \(\varepsilon_{L,\theta}\) is an explicitly defined sign character of \(L^\theta\).