Spherical unitary highest weight representations (Q2759063)

Let \(G\) be a Lie group and \(H\) a closed subgroup. An irreducible unitary representation \((\pi ,{\mathcal H})\) of \(G\) is said to be \(H\)-spherical if \(({\mathcal H}^{-\infty })^H\not=\{O\}\), where \(({\mathcal H}^{-\infty })^H\) is the space of \(H\)-invariant distribution vectors, and then \(({\mathcal H}^{-\infty })^H\) is one dimensional. The authors consider the case of a Hermitian Lie group (this means that \(G\) is simple and the associated Riemannian symmetric space \(G/K\) is Hermitian, where \(K\) is a maximal compact subgroup of \(G\)) and a highest weight representation \((\pi _{\lambda },{\mathcal H}_{\lambda })\). Such a representation is associated to an irreducible representation \((\pi ^K_{\lambda },F(\lambda))\) of \(K\) (\(\lambda \) denotes its highest weight). One assumes that \(G/H\) is a compactly causal symmetric space. If \((\pi _{\lambda },{\mathcal H}_{\lambda })\) is \(H\)-spherical, then \((\pi ^K_{\lambda },F(\lambda))\) is \(H\cap K\)-spherical. The converse is true if \(\lambda \) is regular, but is not true in general if \(\lambda \) is singular. The first main result of this paper is of algebraic nature. Let \(N(\lambda)\) be the generalized Verma module associated to \(\lambda \), and \(L(\lambda)\) its unique simple quotient. Suppose that the reduction level of \(L(\lambda)\) is odd. Then \((\pi _{\lambda },{\mathcal H}_{\lambda })\) is \(H\)-spherical if and only if \((\pi ^K_{\lambda },F(\lambda))\) is \(H\cap K\)-spherical. When \(G/K\) is a Hermitian symmetric space of tube type, i.e. can be realized as a tube over a symmetric cone \(\Omega \), the representation \((\pi _{\lambda },{\mathcal H}_{\lambda })\) can be realized on a Hilbert space of \(F(\lambda)\)-valued distributions supported in \(\overline{\Omega }\). In this realization \(N(\lambda)\) is the space of functions of the form \(f(x)=e^{-\text{tr} (x)} p(x)\), where \(p\) is an \(F(\lambda)\)-valued polynomial, and \(L(\lambda)=R(\lambda)\cdot N(\lambda)\), where \(R(\lambda)\) is an \(\text{End}\bigl(F(\lambda)\bigr)\)-valued distribution. One assumes that \(G/H\) is a symmetric space of Cayley type. The second main result says that, if \((\pi _{\lambda }^K,F(\lambda))\) is \(H\cap K\)-spherical and \(\text{supp} (R(\lambda))\not=\overline{\Omega }\), then \((\pi _{\lambda },{\mathcal H}_{\lambda })\) is \(H\)-spherical if and only if the reduction level is odd.