On a generalization of a result of Howe for unipotent groups (Q6652384)

Let \(G\) be a Lie group and \(H\) a closed subgroup of \(G\). Let \( \chi \) be a unitary character of \(H\) and \(\pi\) an irreducible unitary representation of \(G\). The determination of the subspace formed by the \( \chi \)-semi-invariant vectors of \(\pi\) is an important problem in harmonic analysis because it is linked to the disintegration of the representation \(\mathrm{ind}^{G}_{H}\chi\). Several results related to this topic have been established for different classes of Lie groups in [\textit{R. Howe}, Pac. J. Math. 73, 329--364 (1977; Zbl 0383.22009); \textit{H. Fujiwara}, Pac. J. Math, 329, 352 (1987; Zbl 0588.22008)].\N\NThe paper under review deals with the situation when \(G\) is a \(p\)-adic unipotent Lie group and \(H\) is a normal closed subgroup of \(G\). In this situation, the unitary character of \(H\) is determined by a linear form \(f\) on the Lie algebra \( \mathfrak{g} \) of \(G\) at which the Lie algebra \( \mathfrak{h} \) of \(H\) is subordinate, i.e. \( ( f, [ \mathfrak{h}, \mathfrak{h}] )= 0 \). The space of the representation \( \pi \) is denoted \( \mathcal{H}_{\pi} \), and \( \mathcal{H}_{\pi}^{\infty} \) denotes the subspace of smooth vectors of \( \mathcal{H}_{\pi} \), i.e. vectors which are fixed by an open compact subgroup of \(G\). Further, let \( \mathcal{H}_{\pi}^{-\infty} \) be the space of distribution vectors of \(\pi\), namely, the algebraic dual of \( \mathcal{H}_{\pi}^{\infty} \). The subspace \( \mathcal{H}_{\pi}^{\infty} \) is \(G\)-invariant, and the group \(G\) acts by the contragradient representation, denoted by \( \pi_{-\infty} \), on \( \mathcal{H}_{\pi}^{-\infty} \). The author denotes by \( ( \mathcal{H}_{\pi}^{-\infty} )^{H, \chi} \) the subspace of \( \mathcal{H}_{\pi}^{-\infty} \) formed by vectors \(a\) which are \( \chi \)-semi-invariant, i.e. which satisfy the condition \( \pi_{-\infty} ( h )a = \chi ( h^{-1} )a\) for all \( h\in H \). Now if \(l\) is an element of \( \mathfrak{g}^{*} \), the dual of \( \mathfrak{g} \), then \( \mathfrak{g} ( l ) \) denotes the Lie algebra of the stabilizer of \(l\) under the coadjoint action of \(G\) on \( \mathfrak{g}^{*} \). Also, \( Q ( l , \mathfrak{g} ) \) denotes the set of the subalgebras \( \mathfrak{f} \) of \(\mathfrak{g} \) such that their sum with \( \mathfrak{g} ( l ) \) is a maximal totally isotropic space for \( \beta_{l} \), where \( \beta_{l} \) is the alternating bilinear form on \(\mathfrak{g}\) defined by \( \beta_{l} ( X, Y ) = \langle l, [ X, Y ]\rangle , X, Y\in\mathfrak{g} \). Finally, let \( \mathfrak{h^{\perp}} \) be the space of linear forms on \(\mathfrak{g} \) that vanish on \( \mathfrak{h} \), and let \( \Omega (\pi) \) be the coadjoint orbit in \(\mathfrak{g}^{*} \) corresponding to the irreducible representation \(\pi\). In this notation, the main result of the paper sounds as follows.\N\NMain Theorem. Assume that \( \Omega (\pi)\cap ( f + \mathfrak{h^{\perp}} )\not= \emptyset \). Then\N\[\N\dim ( \mathcal{H}_{\pi}^{-\infty} )^{H, \chi} = \begin{cases} 1 & \mathrm{if} \ \mathfrak{h}\in Q ( l , \mathfrak{g} ),\\\N\infty & \mathrm{if \ not} \end{cases}\N\]\Nfor one and therefore for all \( l\in \Omega (\pi)\cap ( f + \mathfrak{h^{\perp}} ) \).