Standard symmetric operators in Pontryagin spaces: A generalized von Neumann formula and minimality of boundary coefficients (Q1868685)

A symmetric operator \(S\) in a Pontryagin space is used to characterize the boundary coefficients, which are meromorphic matrix functions on \(\mathbb{C}\setminus\mathbb{R}\). If \(Q\) is an invertible self-adjoint matrix, then a \(Q\)-boundary coefficient is a matrix valued function \({\mathcal U}\) which satisfies several conditions including the equality \({\mathcal U}(z)Q^{-1} {\mathcal U}(z^*)^*=0\), and the existence of the limit \[ \lim_{w\to z^*}{{\mathcal U}(z)Q^{-1}{\mathcal U}(w)^* \over z-w^*} \] at each \(z\in\text{dom}({\mathcal U})\). A closed symmetric relation in a \(\kappa\)-Pontryagin space is considered to be standard if for some \(\mu\in \mathbb{C}\setminus \mathbb{R}\), \(\mu\notin\sigma_p (S)\), each of the sets \(\mathbb{C}^\pm\cap \sigma_p(S)\) contains at most \(\kappa\) points. A generalization of the von Neumann's formula \[ S^{\langle *\rangle}=S +S^{\langle *\rangle} \cap\mu I+S^{ \langle *\rangle} \cap\mu^*I,\;\mu\in\mathbb{C} \setminus\mathbb{R}, \] is introduced to study a minimality property of the \(Q\)-boundary coefficients, which is expressed in terms of maximal rank of a matrix formed with the values of \({\mathcal U}\) at \(\kappa\) distinct points in \(\mathbb{C}^+\cap \text{dom} ({\mathcal U})\). The proofs require an extensive knowledge of the theory of operators in indefinite inner product spaces. Detailed explanations, partial results, and references to previous techniques are very well organized in six sections of the paper and an appendix (on Schur functions).