Gelfand pairs with coherent states (Q1282066)

If \(K\) is a compact group acting by automorphism on a locally compact group \(N\), then we say that \((K,N)\) is a Gelfand pair if the algebra of \(K\)-invariant functions on \(N\) is commutative under the convolution. A particular case is when \(K\) is a connected subgroup of \(U(n)\) and \(N=H_n\), where \(U(n)\) is the unitary group and \(H_n\) is the \(2n+1\)-dimensional Heisenberg group. There exists a geometric condition for \((K,H_n)\) to be a Gelfand pair: \((K,H_n)\) is such a pair if and only if the intersection of each coadjoint orbit of the semidirect product \(G\) of \(K\) and \(H_n\) with \((\text{Lie},K)^\bot\) contains at most one integral \(K\)-orbit. Using coherent states, the author defines a generating function of multiplicity \(m_\rho\) in \(\widehat K: m_\rho (r)=\sum _{n=0}^\infty a_nr^n\), where \(a_n = 0,1,2,\dots \) and \(\lim_{r\to 1} m_\rho (r)=\text{mult} (\rho, W_\nu)\) (\(W_\nu\) is the generic representation of \(H_n\) naturally extended to \(G\)). Then \((K,H_n)\) is a Gelfand pair if and only if \(\lim_{r\to 1} m_\rho (r) \leq 1\). It is proved that if \(m_\rho\) is a nonhomogeneous function, then there exist at least two \(K\)-orbits in the intersection of the generic coadjoint orbit associated to \(\rho\) with \((\text{Lie},K)^\bot\).