Unitary colligations in \(\pi _ k\)-spaces, characteristic functions and Shtraus extensions (Q1063219)

Let S be a symmetric subspace (multivalued operator) in \(H^ 2\), where H is a Hilbert space. A family \(\{\) T(\(\ell)|\) \(\ell \in {\mathbb{C}}\cup \{\infty \}\}\) of linear manifolds \(T(\ell)\subset H^ 2\) of the form \[ T(\ell)=\{\{Pf,Pg\}| \quad \{f,g\}\in A,\quad g-\ell f\in H\},\quad \ell \in {\mathbb{C}}, \] \[ T(\infty)=\{\{f,Pg\}| \quad \{f,g\}\in A,\quad f\in H\}, \] where A is a selfadjoint subspace in \(\tilde H^ 2\) with nonempty resolvent set, \(\tilde H\) is a Pontryagin space containing H as a subspace, \(S\subset A\) and P denotes the orthogonal proection of \(\tilde H\) onto H, is called a Štraus extension of S associated with A in \(\tilde H.\) In a previous paper [Integral equations Oper. Theory 7, 459-515 (1984; Zbl 0552.47014)], we characterized T(\(\ell)\) for \(\ell \in {\mathbb{C}}\setminus {\mathbb{R}}\). In this paper we describe T(\(\lambda)\) for \(\lambda\in {\mathbb{R}}\cup \{\infty \}\) in terms of certain boundary values of characteristic functions of a unitary colligation defined by means of the restriction of the Cayley transform of A to the defect subspaces of S. The description is based on a theory of characteristic functions of unitary colligations in Pontryagin spaces [the authors, Characteristic functions of unitary operator colligations in Pontryagin spaces, to be published] analogous to a similar theory in Hilbert spaces presented by M. S. Brodskii. The present paper generalizes and unifies results obtained by A. V. Štraus.