On quasi-positive definite functions and unitary representations of groups in Pontrjagin spaces (Q1254626)

Let \(G\) be a topological group and \(U_n(G)\) \([CU_n(G)]\) be the set of all [cyclic] unitary representations of \(G\) in Pontryagin spaces with negative rank \(n\). For any function \(\Phi(g,h)\) on \(G\times G\) and any \(g_i\in G\) \((1\le i\le m)\) we denote by \(\Phi[g_1,g_2,\ldots,g_m]\) the \(m\)-th order matrix whose \((i,j)\)-coefficient is \(\Phi(g_i,g_j)\). The number of negative eigenvalues of any Hermitian matrix \(H\) is written as \(r_{-}(H)\). We say that a continuous function \(\Phi(g,h)\) on \(G\times G\) is a Hermitian kernel of negative rank \(n\) if the following conditions (a) -- (c) are satisfied: (a) \(\Phi(h,g) = \overline{\Phi(g,h)}\) for any \(g, h\) in \(G\), (b) \(r_{-}(\Phi[g_1,g_2,\ldots,g_m])\le n\) for any \(g_i\in G\) \((1\le i\le m)\), (c) \(r_{-}(\Phi[h_1,h_2,\ldots,h_k])=n\) for some \(h_j\in G\) \((1\le j\le k)\). A continuous function \(\varphi(g)\) on \(G\) is called a quasi-positive definite function of negative rank \(n\) if the function \(\Phi(g,h) = \varphi(h^{-1}g)\) on \(G\times G\) is a Hermitian kernel of negative rank \(n\). The set of all such functions is denoted by \(P_n(G)\). \(P_0(G)\) is regarded as the space of continuous positive definite functions on \(G\). The main results in the paper are as follows: (I) For any \(\{U_g,H\} \in U_n(G)\) and \(f\in H\), the function \(\varphi(g) = \langle U_gf, f\rangle\) is in \(P_m(G)\) for some \(m\), \(0\le m\le n\). Especially if \(f\) is \(U\)-cyclic, then \(\varphi(g)\) is in \(P_n(G)\). (II) For every function \(\varphi(g)\) in \(P_n(G)\) there corresponds \(\{U_g,H\}\) in \(CU_n(G)\) such that \(\varphi(g)\) is given in the form: \(\varphi(g) = \langle U_gf,f\rangle\), where \(f\in H\) is a \(U\)-cyclic vector. These are the natural generalization of the well known correspondence between the space \(P_0(G)\) and the class of all cyclic unitary representations of \(G\) in Hilbert spaces. Moreover, some analogue of \textit{R. Godement}'s article [Trans. Am. Math. Soc. 63, 1--84 (1948; Zbl 0031.35903)] are also considered.