On the Radon transform of \(M_{2,n} ({\mathbf H})\) (Q1908868)

Inspired by an ideal of \textit{C. Meaney} [Ark. Mat. 24, 131-140 (1986; Zbl 0608.43008)], we consider in this paper the Radon transform in the sense of \textit{S. Helgason} [Adv. Math. 36, 297-323 (1980; Zbl 0441.53040)] on the Cartan motion group associated to a Riemannian symmetric space. Let \(G/K = Sp (2,n)/Sp (2) \times Sp (n)\). Let \({\mathfrak g} = {\mathfrak k} + {\mathfrak p}\) be its Cartan decomposition and \(G_0 = K \rtimes {\mathfrak p} \) the Cartan motion group, then \(G_0/K = M_{2,n} (\mathbb{H})\) is the space of all \(2 \times n\) matrices whose elements are quaternion. We first give an explicit expression for the elementary spherical function \(\varphi_\lambda\) on the symmetric space \(Sp (2,n)/Sp (2) \times Sp (n)\), and then follow \textit{J. L. Clerc}'s theorem [Studia Math. 57, 27-32 (1976; Zbl 0335.43010)] \(\varphi_\lambda (H_t) = \lim_{\varepsilon \to 0} \varphi_{{\lambda \over \varepsilon}} (H_{\varepsilon t})\) to obtain the generalized Bessel function on the Cartan motion group \(G_0\). Using the Weyl integral fractional transform we prove that, for any \(Sp(2) \times Sp (n)\)-invariant function \(f\) on \(p\), the function \(F(t) = f(H_t)\) \((t^2_1 - t^2_2)\) satisfies a certain hyperbolic equation (compare with [C. Meaney, op. cit.; corollary 25]; in this case \(f\) can be expressed by its Radon transform). Using Riemann's method of successive approximation, we prove that there exists a unique solution of this hyperbolic equation. As a consequence we derive that every regular \(Sp(2) \times Sp (n)\)-orbit on \(M_{2,n} (\mathbb{H})\) is not the set of synthesis of the Fourier algebra.