Restriction of most degenerate representations of \(O(1,N)\) with respect to symmetric pairs (Q2788593)

K. Kodaira has a great influence on the spectral theory of self-adjoint differential operators. The so-called Weyl-Stone-Kodaira-Titchmarsh theory gives eigenfunction expansions for self-adjoint second order differential operators in one variable. It provides a uniform treatment of classical eigenfunction expansions such as spectral decompositions into Bessel functions, Hermite polynomials, or Laguerre functions. In [Math. Japon. 1, 6--23 (1948; Zbl 0041.29503)], \textit{K. Kodaira} considered a differential operator \(L=\frac{dp}{dx}(x)\frac{d}{dx}+q(x)\) on an interval \((a,b)\subset\mathbb R\). Then, \(L\) extends to a self-adjoint operator on the space of square integrable functions on \((a,b)\) and an expansion into eigenfunctions is of the form: NEWLINE\[NEWLINEu(x)=\sum\limits_{}^{}\int\limits_{-\infty}^\infty s_j(x,\lambda)\int\limits_a^bs_k(y,\lambda)u(y)dyd\rho_{jk}(\lambda),\tag{*}NEWLINE\]NEWLINE where \(s_1(\cdot,\lambda)\) and \(s_2(\cdot,\lambda)\) are linearly independent solutions to the equation \(L u=\lambda u\). The author was able to find an explicit formula for \(d\rho_{jk}\) in terms of the characteristic functions, revealing the explicit relation between the density measures and the asymptotic behavior of eigenfunctions. The formula \((*)\) makes it possible to apply the spectral decomposition theorem to concrete settings and in particular it has a significant impact on the harmonic analysis on Lie groups. For a given variety \(X\) and a Lie group \(G\) acting on it, a fundamental problem in the harmonic analysis on \(X\) is to expand an arbitrary function on \(X\) into joint eigenfunctions for the \(G\)-invariant differential operators on \(X\). An explicit description of such an expansion is called Plancherel Theorem. Plancherel Theorems for reductive homogeneous spaces \(G/H\) can be viewed as induction problems, decomposing the induced representation \(L^2(G/H)=\text{Ind}^G_H(\mathbf{1})\) into irreducible \(G\)-representations. As well as induction problems one may consider restriction problems, that is, to ask how a representation decomposes when restricted to a subgroup. For \(G=O(1,n+1)\), it is known that on the level of \((\mathfrak{g},K)\)-modules all irreducible unitary representations of \(G\) are obtained as subrepresentations of representations induced from a parabolic subgroup \(P=MAN\) that is unique, up to conjugation, and there are group isomorphisms \(M\cong O(n)\times(\mathbb Z/2\mathbb Z)\), \(A\cong\mathbb R^+\), and \(N\cong\mathbb R^n\). The representation \(\pi_{\sigma,\varepsilon}^{O(1,n+1)}\) of \(G\), which is induced from the character of \(P\) given by the character \(\sigma\in\mathbb C\) of \(A\) and the character \(\varepsilon\in\mathbb Z/2\mathbb Z\), is irreducible and unitarizable if and only if \(\sigma\in i\mathbb R\cup(-n,n)\). Using the same symbol for the corresponding irreducible unitary representations, these representations are called unitary principal series representations for \(\sigma\in i\mathbb R\), and complementary series representations for \(\sigma\in(-n,n)\). There are natural isomorphisms \(\pi_{-\sigma,\varepsilon}^{O(1,n+1)}\cong\pi_{\sigma,\varepsilon}^{O(1,n+1)}\) for \(\sigma\in i\mathbb R\cup(-n,n)\). For \(\sigma=n+2u, u\in\mathbb N\), the representation \(\pi_{\sigma,\varepsilon}^{O(1,n+1)}\) has unique non-trivial subrepresentation \(\pi_{\sigma,\varepsilon,\text{sub}}^{O(1,n+1)}\) which is irreducible and unitarizable.NEWLINENEWLINEIn this paper, the authors study the restriction of \(\pi_{\sigma,\varepsilon}^{O(1,n+1)}\) for \(\sigma\in i\mathbb R\cup(-n,n)\), and \(\pi_{\sigma,\varepsilon,\text{sub}}^{O(1,n+1)}\) for \(\sigma\in n+2\mathbb N\), \(\varepsilon\in\mathbb Z/2\mathbb Z\), with respect to any symmetric pair \((G,H)\), where the subgroup \(H\) is conjugate to \(O(1,m+1)\times O(n-m)\). They find complete branching law for this restriction. Since the unitary representations considered here are induced from a character of a parabolic subgroup or its irreducible quotient, the authors show that they belong either to the unitary spherical principal series, to the spherical complementary series, or to discrete series for the hyperboloid. In the case of \(0 < m < n\) the decomposition consists of a continuous part and a discrete part, where the continuous part is given by a direct integral of unitary principal series representations whereas the discrete part consists of finitely many representations which either belong to the complementary series or are discrete series for the hyperboloid. Moreover, the authors compute the explicit Plancherel formula on the Fourier transformed side of the non-compact realization of the representations by using the spectral decomposition of a certain hypergeometric type ordinary differential operator.