On extreme points of families of analytic functions with values in a convex set (Q1082480)

Let \(\Delta\) denote the unit disc \(\{z: | z| <1\}\) and D denote a convex domain in the complex plane which contains 0 and which is not the whole plane. Let F denote the family of all analytic functions f on \(\Delta\) for which \(f(\Delta) \subset D\) and \(f(0)=0\). For such functions \(f(\zeta)=\lim_{r\to 1}f(r\zeta)\) exists for almost all \(\zeta\in \partial \Delta\). \textit{Y. Abu-Muhanna} and \textit{Th. MacGregor} [Math. Z. 176, 511-519 (1981; Zbl 0461.30018)] considered the problem of determining the set E(F) of extreme points of F. It follows from a result of \textit{D. J. Hallenbeck} and \textit{Th. H. MacGregor} [Pac. J. Math. 50, 455-468 (1974; Zbl 0258.30015)] that if D is not a half plane, then \(\{f\in F:\) \(f(\zeta)\in \partial D\) for almost all \(\zeta\in \partial D\}\subset E(F)\). In the case D is a strip, \textit{J. G. Milcetich} [Proc. Am. Math. Soc. 45, 223-228 (1974; Zbl 0296.30015)] proved that the previous inclusion is actually an equality. \textit{Y. Abu-Muhanna} and \textit{Th. H. MacGregor} [Math. Z. 176, 511-519 (1981; Zbl 0461.30018)] proved the inclusion is an equality whenever D is a wedge. They conjectured this equality holds when D is any (bounded or unbounded) convex polygon. In this paper, the author proves that the conjecture is false. He first proves that if D is not a half plane, a strip or a wedge then there exists \(f\in E(F)\) for which \(f(\zeta)\not\in \partial D\) on a subset of \(\partial \Delta\) of positive measure. A second example is then constructed so that \(f\in E(F)\) and \(f(\zeta) \not\in \partial D\) for almost all \(\zeta\in \partial \Delta\). The measure theoretic considerations in this second construction are involved and clever. Both constructions depend in an essential way on properties of the Poisson integral formula. The problem remains to characterize the set E(F) when D is a convex polygon. At this point, it is not clear how to formulate a reasonable conjecture. This excellent paper is commended to the attention of those interested in problems on the interface between complex and real analysis.