Invariant trilinear forms for spherical degenerate principal series of complex symplectic groups (Q2788775)

The author evaluates an integral \({\mathcal I}_n\) on the product of three \(N\)-dimensional unit spheres. This integral generalizes Bernstein-Reznikov integrals which are integrals on three unit circles. Such an integral is related to an invariant trilinear form, i.e., a trilinear map \(\Psi :V_1\times V_2\times V_2 \to {\mathbb C}\) such that NEWLINE\[NEWLINE\Psi \bigl(\pi _1(g)v_1,\pi _2(g)v_2,\pi _3(g)v_3\bigr) =\Psi (v_1,v_2,v_2),NEWLINE\]NEWLINE where \(\pi _1\), \(\pi _2\), \(\pi _3\) are representations of a group \(G\) on the vector spaces \(V_1\), \(V_2\), \(V_3\). In fact \({\mathcal I}_n\) is the value \(\Psi (v_1,v_2,v_2)\) of the trilinear form for special vectors \(v_1\), \(v_2\), \(v_3\). In this paper \(\pi _1\), \(\pi _2\), \(\pi_3\) are spherical degenerate principal series representations of the complex symplectic group \(G=Sp(n,{\mathbb C})\). These representations can be realized on \(L^2(S^{4n-1})\), and \(v_1=v_2=v_3\) is a constant function. The integral \({\mathcal I}_n\) turns out to be the trace of the product of three Knapp-Stein intertwining operators. When restricted to the maximal compact group \(K=Sp(n)\), these representations decompose multiplicity free. Hence \({\mathcal I}_n\) can be developed as a series involving the eigenvalues of the intertwiners. By a careful study of this series, the author establishes that \({\mathcal I}_n\) is, up to a gamma factor, the value at 1 of a hypergeometric function \(_6F_5\).