Riesz potentials and integral geometry in the space of rectangular matrices (Q2504379)

Let \(k,m,n\) be positive integers, with \(0<k<n\), \(\mathfrak{M}_{n,m}\) the space of rectangular matrices and \(V_{n,n-k}\) the Stiefel manifold of orthonormal \((n-k)\)-frames in \(\mathbb{R}^n\). For every \(\xi\in V_{n,n,-k}\) and \(t\in\mathfrak{M}_{n-k,m}\), define a linear manifold \(\tau=\tau(\xi,t)\) (a matrix \(k\)-plane in \(\mathfrak{M}_{n,m}\)) by setting \[ \tau(\xi,t)=\{x\in\mathfrak{M}_{n,m}:\xi'x=t\}, \] (\(\xi'\) is the transpose of \(\xi\)). Let \(\mathfrak{T}\) be the variety of all such planes. The matrix \(k\)-plane Radon transform assigns to a function \(f\) defined on \(\mathfrak{M}_{n,m}\) the collection of integrals \[ \widehat{f}(\tau)=\int_{x\in\tau}f(x),\qquad\tau\in\mathfrak{T}. \] The author develops several important analytic tools necessary to investigate this class of Radon transforms. In a quite self-contained way, he estabilishes foundational material on higher rank harmonic analysis and the related Riesz potentials and Radon transforms. We overview the main results. Denote by \(\partial\) the \(n\times m\) matrix \((\partial/\partial x_{ij})\), where \(x=(x_{ij})\in\mathfrak{M}_{n,m}\). The Cayley-Laplace operator on \(\mathfrak{M}_{n,m}\) is defined by setting \(\Delta=\det (\partial'\partial)\) [see also \textit{S. P. Khekalo}, Dokl. Math. 63, No. 1, 35--37 (2001); translation from Dokl. Akad. Nauk, Ross. Akad. Nauk 376, No. 2, 168--170 (2001; Zbl 1047.31006)]. The polar decompositon for \(x\in\mathfrak{M}_{n,m}\) is: \(x=vr^{1/2}\), where \(v\in V_{n,m}\) and \(r=x'x\in\mathcal{P}_m\), (this is the open convex cone of positive definite matrices). In the third section of the paper, the author extends to \(\Delta\) the classical formula for the radial part of the Laplace operator. Set \(D=\det[((1+\delta_{ij})/2)\partial/\partial r_{ij}]\) (this acts on \(\mathcal{P}_m\)) and \[ L=4^m | r| ^{d-n/2}D| r| ^{n/2-d+1}D,\qquad d=(m+1)/2, \] where \(| a| \) is the absolute value of \(\det a\) if \(a\) is a square matrix. If \(n\geq m\geq 1\), \(\Omega\subseteq\mathfrak{M}_{n,m}\) is an open set consisting of matrices of rank \(m\) and \(f\) is of the form \(f(x)=f_0(x'x)\), with \(f_0(r)\in C^{2m}(\mathcal{P}_m)\), then for \(x\in\Omega\) we have: \[ (\Delta f)(x)=(Lf_0)(x'x). \] A consequence is the following identity of Bernstein type [\textit{I. N. Bernstein}, Funct. Anal. Appl. 6, 273--285 (1972); translation from Funkts. Anal. Prilozh. 6, No. 4, 26--40 (1972; Zbl 0282.46038)]: if \(| x| _m=\det(x'x)^{1/2}\) then \[ \Delta| x| ^\lambda_m=\mathcal{B}(\lambda)| x| _m^{\lambda-2}, \] where \(\mathcal{B}(\lambda)=(-1)^m\prod_{i=0}^{m-1}(\lambda+i)(2-n-\lambda+i)\). In the fourth section, the author introduces the zeta integral \[ \mathcal{Z}(f,\alpha-n)=\int_{\mathfrak{M}_{n,m}}f(x)| x| ^{\alpha-n}_mdx \] where \(f\) is a Schwartz function on \(\mathfrak{M}_{n,m}\) and \(n\geq m\). Using the Bernstein type identity, he proves the following functional equation: \[ \frac{\mathcal{Z}(f,\alpha-n)}{\Gamma_m(\alpha/2)}=\pi^{nm/2}2^{m(\alpha-n)} \frac{\mathcal{Z}(\mathcal{F}f,-\alpha)}{\Gamma_m((n-\alpha)/2)}; \] in this formula \(\Gamma_m\) is the Siegel gamma function associated to \(\mathcal{P}_m\) and \(\mathcal{F}f\) is the Fourier transform of \(f\). He also investigates the normalized zeta distribution \[ (\zeta_\alpha,f)= \text{a.c. }\frac{1}{\Gamma_m(\alpha/2)}\int_{\mathfrak{M}_{n,m}} f(x)| x| ^{\alpha-n}_m,\qquad f\in\mathcal{S}(\mathfrak{M}_{n,m}), \] where ``a.c.'' means analytic continuation. The corresponding Wallach set \(\mathcal{W}\) is given by all \(\alpha\) for which \(\zeta_\alpha\) is a positive measure. Each \(\alpha>m-1\) belongs to \(\mathcal{W}\), but the author proves that also the integers \(\alpha=0,1,\dots, m-1\) belong to \(\mathcal{W}\), obtaing explicit expression of \(\zeta_\alpha\) for these values of \(\alpha\). He also conjectures that \(\mathcal{W}=\{0,1,\dots,m-1\}\cup\{\alpha>m-1\}\). In the fifth section, he investigates the Riesz potential associated to the zeta function, defined by setting \[ (I^\alpha f)(\alpha)=\frac{1}{\gamma_{n,m}(\alpha)}\int_{\mathfrak{M}_{n,m}}f(x-y)| y| ^{\alpha-n}_mdy \] where \[ \gamma_{n,m}(\alpha)=\frac{2^{\alpha m}{\pi^{nm/2}{\Gamma_m(\alpha/2)}}}{\Gamma_m((n-\alpha)/2)},\qquad\alpha\neq n-m+1,n-m+2,\dots \] Using the matrix modification of the Gauss-Weierstrass integral \[ (W_t f)(x)=\int_{\mathfrak{M}_{n,m}}h_t(x-y)f(y)dy,\qquad t\in\mathcal{P}_n, \] where \(h_t\) is the heat kernel on \(\mathcal{P}_n\), he represents the Riesz potential in the form \[ (W_t[I^\alpha f])(x)=(I_-^{\alpha/2}g_x)(f),\qquad g_x(t)=(W_tf)(x), \] for \(m-1<Re\; \alpha<n-m+1\), where \(I_-^\lambda\) is the Gårding-Gindikin fractional integral on \(\mathcal{P}_m\). In the last section, the author applies the topics previously developed to the study of the matrix \(k\)-plane Radon transform. He proves a Fuglede formula: for \(f\in L^p(\mathfrak{M}_{n,m}),1\leq p\leq n(k+m-1)\) \[ (\widehat{f})^\vee(x)=\text{ const} (I^kf)(x), \] where the left side is the mean value of \(\widehat{f}(\tau)\) over all matrix \(k\)-planes passing through \(x\). He also introduces a space \(\Phi'\) of distributions of the \textit{V. I. Semyanistyj} type [Sov. Math., Dokl. 2, 59--62 (1961); translation from Dokl. Akad. Nauk SSSR 136, 288--291 (1961; Zbl 0100.36602)] and proves an inversion formula in the \(\Phi'\)-sense. The paper ends with an appendix with detailed proofs of some identities for the beta and gamma functions on \(\mathcal{P}_n\).