A note on degenerate Whittaker models for general linear groups (Q2288309)

Let \(F\) be non-Archimedean local field, that is, a finite extension of the valued field \(p\)-adic numbers \(\mathbb{Q}_p\) or the function field \(\mathbb{F}_p((t))\). Let \(G = \mathrm{GL}_n(\mathbb{F})\) be the general linear group of invertible \(n \times n\)-matrices with entries in \(\mathbb{F}\). An action (or representation) of \(G\) on a complex vector space \(V\) is \textit{smooth} if for every \(v\) in \(V\) there is an open subgroup of \(G\) that fixes \(v\). Of particular interest are the (smooth) Speh representations as the fundamental unitary smooth representations: every irreducible unitary representation of \(G\) is obtained by (successively parabolically) inducing representations starting from either a tensor product of Speh representations or a complementary series obtained from a tensor product of two copies of the same Speh representation. By a well-known formula for the character of a (parabolically) induced representation, expressed by the character of the induced representation, the characters of irreducible unitary representations of \(G\) can be computed from the characters of the Speh representations [\textit{G. Chenevier} and \textit{D. Renard}, Represent. Theory 12, 447--452 (2008; Zbl 1163.22008)]. Let \(U\) be the subgroup of upper triangular unipotent matrices in \(G\). Every character \(\Theta\) of \(U = (u_{i,j})\) is of the form \[ u \mapsto \psi(a_1 u_{1,2} + \cdots + a_{n-1} u_{n-1,n}) \] for a character \(\psi \colon \mathbb{F} \to \mathbb{C}^*\) and \(a_1, \dots, a_{n-1}\) in \(\mathbb{F}\). Let \(\mathcal{S} = \mathcal{S}(\Theta)\) denote the subset of all \(i\) in \(\{ 1,\dots, n-1 \}\) such that \(a_i = 0\). The character is \textit{non-degenerate} if \(\mathcal{S} = \emptyset\). A smooth irreducible action \(\pi\) of \(G\) \textit{admits a degenerate Whittaker model} if there is a degenerate character \(\Theta\) of \(U\) and a nonzero \(\mathbb{F}[U]\)-linear homomorphism between \(\pi\) and \(\Theta\). For a Speh representation \(\pi\) of \(G\), this concise article gives all the subsets \(\mathcal{S}\) of \(\{ 1,\dots, n-1 \}\) for which there is a degenerate Whittaker model of \(\pi\) such that \(\mathcal{S} = \mathcal{S}(\Theta)\). More exactly, if \(\pi\) is \begin{itemize} \item[--] parabolically induced from an irreducible essentially square integrable representation of \(\mathrm{GL}_a(\mathbb{F})\), and \item[--] supported on the \textit{cuspidal line} of the irreducible representation \(\rho\) of \(\mathrm{GL}_d(\mathbb{F})\) (that is, \(\pi\) is a subrepresentation of a product of \(|\det|_F^i \rho\) for certain integers \(i\)), \end{itemize} then there is a degenerate Whittaker model for \(\mathcal{S}\) if and only if \begin{itemize} \item[--] all \(i\) in \(\mathcal{S}\) are divisible by \(d\), and \item[--] \(\mathcal{S}\) contains \(ld, \dots, (k-1)ld\) where \(l = \frac{a}{d}\) and \(k = \frac{n}{a}\). \end{itemize} To this end, besides combinatorial observations, a necessary and sufficient condition for the existence of a non-degenerate Whittaker model is studied on the Jacquet module of \(\pi\) (the largest quotient on which the unipotent radical of a standard parabolic subgroup acts trivially), and of which an explicit description is given in [loc. cit.].