Représentations monomiales des groupes de Lie nilpotents. (Monomial representations of nilpotent Lie groups) (Q1073193)

In this paper, we study by the method of orbits due to \textit{A. A. Kirillov} [Usp. Mat. Nauk 17, No.4 (106), 57-110 (1962; Zbl 0106.250)] monomial representations with finite multiplicities of nilpotent Lie groups. For a unitary representation \(\sigma\) of a Lie group, we denote by \({\mathcal H}_{\sigma}\) its Hilbert space and by \({\mathcal H}_{\sigma}^{\pm \infty}\) the space of \(C^{\pm \infty}\)-vectors for \(\sigma\). Let G be a connected and simply connected nilpotent Lie group, H an analytic subgroup of G and let \(\chi\) be a unitary character of H. We consider the induced representation \(\tau =ind^ G_ H \chi\) of G and its canonical central decomposition \(\tau =\int_{\hat G}^{\oplus} m(\pi)\pi d\nu (\pi)\), \(\hat G\) being the unitary dual of G, with a certain Borel measure \(\nu\) on \(\hat G\) and multiplicity function m: \(\hat G\ni \pi \mapsto m(\pi)\in {\mathbb{Z}}_+\cup \{+\infty \}.\) According to this decomposition, an element \(a_{\tau}\in {\mathcal H}_{\tau}^{-\infty}\) given by \(a_{\tau}: {\mathcal H}_{\tau}^{+\infty}\ni \phi \mapsto \overline{\phi (e)}\in {\mathbb{C}}\) (e : the unit of G) is decomposed into \(a_{\tau}=\int_{\hat G}^{\oplus} (\sum^{m}_{k=1}\oplus a^ k_{\pi}) d\nu (\pi)\) [Penney's abstract Plancherel formula, see \textit{R. Penney}, J. Funct. Anal. 18, 177-190 (1975; Zbl 0305.22016)] with some \(a^ k_{\pi}\in {\mathcal H}_{\pi}^{-\infty}.\) Assuming m(\(\pi)\) finite almost everywhere for \(\nu\), we give these \(a^ k_{\pi}\) explicitly to write down a concrete Plancherel formula for \(\tau\).