Annihilators of Frobenius vectors: algebraic structures (Q2900197)

Let \(G=\exp{\mathfrak g}\) be a connected and simply connected nilpotent Lie group with Lie algebra \({\mathfrak g}\) and \(H= \exp{\mathfrak h}\) an analytic subgroup of \(G\) with Lie algebra \({\mathfrak h}\). Let \(\tau\) be a monomial representation induced up from a unitary character \(\chi\) of \(H\). Then \(\chi\) is written as \(\chi(\exp X)= e^{if(X)}\) \((X\in{\mathfrak h})\) with some linear form \(f\) on \({\mathfrak g}\) satisfying \(f([{\mathfrak h},{\mathfrak h}])= \{0\}\). Let \(\widehat G\) be the unitary dual of \(G\) and NEWLINE\[NEWLINE\tau= \int^{\oplus}_{\widehat G} m(\pi)\pi d\mu(\pi)NEWLINE\]NEWLINE the canonical central decomposition of \(\tau\) with some Borel measure on \(\widehat G\), where we suppose that the multiplicities \(m(\pi)\) are finitely bounded. Let \({\mathcal H}^\infty_\tau\) be the space of \(C^\infty\) vectors of \(\tau\) and \({\mathcal H}^{-\infty}_\tau\) the anti-dual of \({\mathcal H}^\infty_\tau\). We put NEWLINE\[NEWLINE({\mathcal H}^{-\infty}_\tau)^{H,\chi}= \{a\in{\mathcal H}^{-\infty}_\tau: \tau'(h)a= \chi(h) a,\,h\in H\},NEWLINE\]NEWLINE where \(\tau'\) denotes the contragredient representation of \(\tau\). Then, the Dirac map \(\delta_\tau:{\mathcal H}^\infty_\tau\to \mathbb{C}\), \(\delta_\tau(\psi)= \overline{\psi(e)}\), \(e\) being the unit element of \(G\), belongs to \(({\mathcal H}^{-\infty}_\tau)^{H,\chi}\). Now, Penney's abstract Plancherel formula for \(\tau\) is written as NEWLINE\[NEWLINE\delta_\tau= \int^{\oplus}_{\widehat G} \Biggl(\sum^{m(\pi)}_{k=1} a^k_\pi\Biggr)\,d\mu(\pi),NEWLINE\]NEWLINE where the vectors \(a^k_\pi\in({\mathcal H}^{-\infty}_\tau)^{H,\chi}\) \((1\leq k\leq m(\pi))\) are the so-called Frobenius vectors (cf. [\textit{R. Penney}, J. Funct. Anal. 18, 177--190 (1975; Zbl 0305.22016)]).NEWLINENEWLINE Let \(D_r\) be the algebra of \(G\)-invariant differential operators on the line bundle with base space \(G/H\) associated to the data \((G,H,\chi)\), and \(C^H_\tau\) the algebra of all \(\text{Ad}^*(H)\)-invariant rational functions on the affine subspace \(f+{\mathfrak h}^\perp\) of \({\mathfrak g}^*\).NEWLINENEWLINE The author considers the map \(\phi: D_\tau\to C^\infty_\tau\), \(D\to\phi(D)=: \phi_D\) introduced by \textit{L. Corwin} and \textit{F. P. Greenleaf} [Commun. Pure Appl. Math. 45, No. 6, 681--748 (1992; Zbl 0812.43004)], which turns out to be a homomorphism, and puts \(\phi^H_\tau:= \text{Im}(\phi)\). Let \(\mu'\) be a homomorphism of \(D_\tau'\) into the algebra of derivations of \(\phi^H_\tau\). The author studies the external semidirect product \({\mathcal E}^H_\tau:= D_\tau'\oplus_{\mu'} \phi^H_\tau\) and its subalgebra of the Frobenius vectors annihilators.