On some degenerate principal series representations of \(O(p,2)\) (Q2718658)

In this paper a degenerate principal non-unitary series of representations of the pseudo-orthogonal group \(G=O(p,2)\), \(p>4\), is studied. These representations are induced by characters (one-dimensional representations) of a maximal parabolic subgroup \(S\) of \(G\). The group \(G\) has only two (up to conjugations) maximal parabolic subgroups, namely, this \(S\) and another, say, \(T\). The latter gives the series of representations occuring in harmonic analysis on hyperboloids on which \(G\) acts. These subgroups \(S\) and \(T\) can be obtained as follows. Let \({\mathfrak g}={\mathfrak k}+{\mathfrak p}\) be the Cartan decomposition of the Lie algebra \({\mathfrak g}\) of \(G\). Let \({\mathfrak a}_{\mathfrak p}\) be a maximal Abelian subalgebra of \({\mathfrak p}\). It has dimension 2. Let us take coordinates \(t_1,t_2\) in \({\mathfrak a}_{\mathfrak p}\) such that (non-zero) roots of the pair \({\mathfrak g}\), \({\mathfrak a}_{\mathfrak p}\) are functions \(\pm t_1\pm t_2\), \(\pm t_1,\pm t_2\). The subgroups \(S\) and \(T\) are the parabolic subgroups whose split components are the subalgebras \({\mathfrak a}=\{t_1 =t_2\}\) and \({\mathfrak b}= \{t_2=0\}\) of \({\mathfrak a}_{\mathfrak p}\), respectively. The character of \(S\) is parametrized by a complex number \(c\). The author denotes the corresponding representations of \(G\) by \(S^c(\mathbb{X}^+)\). They act on functions on the Stiefel manifold \(SO(p)/SO(p-2)\) satisfying a homogeneity condition. The main tool for studying these representations is the restriction to the maximal compact subgroup \(K=O(p) \times O(2)\). Fortunately, this restriction turns out to be multiplicity-free. The \(K\)-types are labelled by pairs \((l_1, l_2)\) of integers with \(l_1\geq l_2\geq 0\). The author succeeded (by means of impressive computations) in determining explicitly the coefficients linking adjacent \(K\)-types. They contain barrier functions. So that, drawing barriers on the plane with coordinates \(l_1,l_2\), we get a quite apparent picture for the structure of representations in question [as, for example, in the paper by the reviewer in Math. USSR, Sb. 10, No. 3, 333-347 (1970); translation from Mat. Sb., N. Ser. 81(123), 358-375 (1970; Zbl 0219.22015) for representations of \(SO_0(p,q)\) associated with a cone]. Finally, the unitarizability of these representations and their subfactors is investigated.