Homogeneous quantization and multiplicities of group representations (Q809439)

Let \(\Phi: X\to {\mathfrak g}^*\) be the momentum mapping for a Hamiltonian action of the Lie group G on the symplectic manifold X. If \(G_{\mu}\subseteq G\) denotes the isotropy group of \(\mu\in {\mathfrak g}^*\) under the coadjoint representation, the quotient space \(\Phi^{- 1}(\mu)/G_{\mu}=X_{\mu}\) is called the reduced space of X at \(\mu\). If \(\mu\) is a clean value of \(\Phi\) and \(G_{\mu}\) acts without change of orbit types on \(\Phi^{-1}(\mu)\), then \(X_{\mu}\) is a symplectic manifold. There is a sense in which \(X_{\mu}\) represents the ``intertwining operators'' from the homogeneous Hamiltonian G space \(G/G_{\mu}\) to X, so that if the whole setup is quantized, \(X_{\mu}\) should represent the ``multiplicity'' of the ``irreducible representation'' \(G/G_{\mu}\) in X. To get a numerical estimate of the ``size'' of a compact symplectic manifold X, the authors use the Riemann-Roch number \(\int_{X}\tau e^{\omega}\), where \(\tau\) represents the Todd class of underlying almost complex structure (well defined up to homotopy), and \(\omega\) is the symplectic structure. If \(G/G_{\mu}\) is an integral coadjoint orbit, it may be quantized by the representation \(\rho_{\mu}\) of G on the holomorphic sections of a line bundle (Borel-Weil-Bott theorem), and the Riemann-Roch theorem says that the dimension of this space of sections equals the Riemann-Roch number of \(G/G_{\mu}.\) Now let \(\rho\) be a unitary representation of G which is intended to quantize the Hamiltonian G space X. If \(G/G_{\mu}\) is an integral coadjoint orbit, we say that the multiplicity principle holds for \(\mu\) if the multiplicity {\#}(\(\rho_{\mu},\rho)\) of \(\rho_{\mu}\) in \(\rho\) equals the Riemann-Roch number of the reduced space \(X_{\mu}.\) In an earlier paper [Invent. Math. 67, No.3, 515-538 (1982; Zbl 0503.58018)] the authors proved the multiplicity principle in the general setting where X is compact and \(\rho\) is the quantization by holomorphic sections of a line bundle. The principle was also proven for so-called homogeneous quantizations of \(S^ 1\) actions by \textit{L. Boutet de Monvel} and the first author [The spectral theory of Toeplitz operators (1981; Zbl 0469.47021)]. In the paper under review, the authors prove the following theorem: Let M be a compact manifold, G a compact Lie group acting on \(T^*M\setminus \{0\}\) by homogeneous canonical transformations, \(\Phi: T^*M\setminus \{0\}\to {\mathfrak g}^*\setminus \{0\}\) the homogeneous momentum mapping. Let \(\rho\) be a representation of G on \(L^ 2(M)\) by Fourier integral operators associated with the given canonical transformations. Let \(V\subseteq {\mathfrak g}^*\setminus \{0\}\) be a closed conic subset consisting of regular elements which are regular values of \(\Phi\) such that, for each \(\mu\in V\), \(G_{\mu}\) acts freely on \(\Phi^{-1}(\mu)\). Then the multiplicity principle is true for all but finitely many integral elements of V. The proof involves a reduction to the case of homogeneous quantizations of \(S^ 1\) actions corresponding to ladder representations. A main step is to show that both the Riemann-Roch numbers and representation multiplicities involved in the principle are given, off a finite subset of V, by polynomials in \({\hat \mu}\). On the technical side, good doses of Fourier integral operator theory and representation theory are involved.