Theta lifting of two-step nilpotent orbits for the pair \(O(p,q)\times Sp(2n,\mathbb{R})\) (Q2751523)

Let \(G\) be a linear reductive real Lie group, \(K\subset G\) the maximal compact subgroup, \(\mathfrak g_0 = \mathfrak k_0\oplus\mathfrak s_0\) the Cartan decomposition of the Lie algebra \(\mathfrak g_0\) of \(G\), and \(\mathfrak g = \mathfrak k\oplus\mathfrak s\) the complexification of this decomposition. Then the complexification \(K_{\mathbb C}\) of \(K\) acts on \(\mathfrak s\) by the adjoint representation, and nilpotent orbits of \(K_{\mathbb C}\) are, by definition, the orbits in \(\mathfrak s\) consisting of nilpotent (in \(\mathfrak g\)) elements. In the paper, the dual pair (in the sense of Howe) \((G,G') = (O(p,q),Sp(2n,\mathbb R))\) is considered. Let \(K_{\mathbb C}, K'_{\mathbb C}\) denote the complexified maximal compact subgroups of \(G, G'\) and \(\mathfrak s, \mathfrak s'\) the complexified Cartan complements. Then the space of matrices \(W = M_{p+q},n(\mathbb C)\) admits two natural equivariant mappings \(\varphi: W\to\mathfrak s\) and \(\psi: W\to\mathfrak s'\). A nilpotent orbit \(\mathcal O\subset\mathfrak s\) is called the theta lift of a nilpotent orbit \(\mathcal O'\subset\mathfrak s'\) if \(\overline{\mathcal O} = \varphi(\psi^{-1}(\overline{\mathcal O'}))\). The main results of the paper are as follows. Assuming that \(2n < \min(p,q)\), the theta lifts of 2-step nilpotent orbits of \(\mathfrak s'\) are found. For some of these theta lifts \(\mathcal O\), the \(K_{\mathbb C}\)-module structure on the ring of regular functions on \(\overline{\mathcal O}\) is described. As an application, a relation between branching coefficients for some classical groups is found.NEWLINENEWLINEFor the entire collection see [Zbl 0964.00042].