Covariants of Sp\(_{n}(\mathbb C)\) and degenerate principal series of GL\(_{n}(\mathbb H)\) (Q2464867)

Let \(m\leq n\) be positive integers. Denote by \(\mathcal{P}(M_{2n,m})\) the algebra of polynomial functions on the space \(M_{2n,m}\) of \(2n\times m\) complex matrices, which carries an action of the group \(GL_{2n}\times GL_m=GL_{2n}(\mathbb{C})\times GL_m(\mathbb{C})\) by \[ [(g,a)f](T)=f(g^t T a),\quad f\in \mathcal{P}(M_{2n,m}),\,g\in GL_{2n},\, a\in GL_m,\,T\in M_{2n,m}. \] Denote by \(\mathcal{R}_{n,m}\) the invariants with respect to the subgroup \(U_{Sp_n}\times SL_m\), which is a subalgebra of \(\mathcal{P}(M_{2n,m})\), where \(U_{Sp_n}\) is the maximal unipotent subgroup of \(Sp_n\), and \(Sp_n\subset GL_{2n}\) is the symplectic group of rank \(n\). The first main result of the paper is an explicit description of \(\mathcal{R}_{n,m}\). The author proves that it is always a polynomial algebra, with free generators \(\eta_0,\eta_2,\cdots,\eta_m\), for \(m\) even, and \(\eta_1,\eta_3,\cdots,\eta_m\), for \(m\) odd. The functions \(\eta_i\) are explicitly given as pfaffians. Now assume that \(m=2k\) is even. Let \(G=GL_n(\mathbb{H})\) be the quaternionic general linear group, and let \(P_k\) be the maximal parabolic subgroup of \(G\) with Levi component \(L=GL_{n-k}(\mathbb{H})\times GL_k(\mathbb{H})\). For \(\alpha\in \mathbb{C}\), let \(\chi_\alpha\) be the character of \(P_k\) which is trivial on the unipotent radical, and is given by \((a,b)\mapsto det(b)^\alpha\) on \(L\). The author then uses the description of \(\mathcal{R}_{n,m}\) to study the degenerate principle series representation \(I(\alpha)=Ind_{P_k}^G \chi_\alpha\). He determines all subrepresentations of \(I(\alpha)\), as well as unitarizability of all its irreducible constituents. Similar families of representations of \(GL_n(\mathbb{C})\) and \(GL_m(\mathbb{R})\) were studied earlier by \textit{R. Howe} and the author [J. Funct. Anal. 166, No.2, 244-309 (1999; Zbl 0941.22016)].