The trace formula for transversally elliptic operators on Riemannian foliations (Q2736045)

The contribution of the paper is a generalization to foliations of an important trace formula of \textit{J. J. Duistermaat} and \textit{V. W. Guillemin} [Invent. Math. 29, 39-79 (1975; Zbl 0307.35071)]. The trace formula of the paper holds for a pseudodifferential operator \(P=\sqrt{A}\), where \(A\) is a positive self-adjoint second order transversally elliptic pseudodifferential operator on the space of smooth half-densities on a compact foliated manifold \((M,{\mathcal F})\). Thus \(A\) may not be elliptic. The typical example is \(A=I+\Delta_H\), where \(\Delta_H\) is the transverse laplacian of a bundle-like metric when \(\mathcal F\) is a riemannian foliation. Another new ingredient, related with noncommutative geometry, is that this trace formula depends on any compactly supported smooth leafwise half-density \(k\) on the holonomy groupoid, whose usual action on the Hilbert space of \(L^2\)-half-densities on \(M\) is denoted by \(R(k)\). NEWLINENEWLINENEWLINEThe author shows that, for every \(f\in C^\infty_c({\mathbb R})\), the operator \(R(k)\int f(t)e^{itP} dt\) is of trace class, and its trace depends continuously on \(f\), defining a distribution \(\theta_k\) on \(\mathbb R\). The trace formula describes the singularities of \(\theta_k\) by using the adaptation to this setting of the geometric information involved in the usual trace formula. In particular, the author defines the transverse bicharacteristic flow \(f_t\) of \(A\) on the conormal bundle of \(\mathcal F\). The corresponding relative fixed point set and relative period set is defined in the usual way ``modulo holonomy''; indeed, dependence on \(k\) is introduced by considering only the holonomy induced by the support of \(k\). Then \(\theta_k\) is shown to be smooth outside the relative period set. For a relative period \(t\), assuming that the corresponding relative fixed point set is clean, \(\theta_k\) is described around \(t\) by using asymptotic expansions and Maslov indices.