Spectrum and multiplicities for restrictions of unitary representations in nilpotent Lie groups (Q1093021)

Let K be a Lie subgroup of the connected, simply connected nilpotent Lie group G, and let \(\pi\) \(\in \hat G\). The main result of the paper is a theorem describing the direct integral decomposition of \(\pi\) /K into irreducibles. Here is a brief account of the statement. Let \({\mathfrak g,k}\) be the corresponding Lie algebras, let \({\mathfrak g^ *,k^ *}\) be their duals, let \(P:= {\mathfrak g^ *}\to {\mathfrak k}^ *\) be the canonical projection, and let \({\mathfrak o}_{\pi}\subset {\mathfrak g}^ *\) be the Kirillov orbit corresponding to \(\pi\). Almost all Kirillov orbits \({\mathfrak o}_{\sigma}\subset {\mathfrak k*}\) meeting P(\({\mathfrak o}_{\pi})\) have the same dimension. The direct integral is over the corresponding representations \(\sigma\) \(\in K{\hat{\;}}\), and the multiplicity \(m_{\sigma}\) of \(\sigma\) is the number of \(Ad^ *(K)\)-orbits in \(P^{-1}({\mathfrak o}_{\sigma})\cap {\mathfrak o}_{\pi}\). The measure also has a simple description. The numbers \(m_{\sigma}\) are (almost everywhere) finite or all infinite; they are a.e. finite iff \(\dim {\mathfrak o}_{\pi}+\dim {\mathfrak o}_{\sigma}=2 \dim Ad^ *(K)\ell\) for generic \(\ell \in {\mathfrak o}_{\sigma}\). The paper also contains some examples, including a calculation of the decomposition of a tensor product.