Notes on \(L(D)\)-convex sets (Q1900725)

The author studies conditions for an open set \(G\) in \(\mathbb{R}^n\) to be \(P\)-convex, where \(P\) is a linear differential operator \(P(D)= \sum_{|\sigma|\leq r} a_\sigma(x) D^\sigma\) with smooth coefficients. First, suppose \(P(D)\) be with constant coefficients. Then the author proves a necessary condition for \(G\) for not being \(P\)-convex; more precisely if \(G\) is not \(P\)-convex then there exists \(u\in {\mathcal B}_{kP^\sim}(G)\cap {\mathcal E}'\) and \(f\in B^c_k(G)\) such that \(Lu= f\) and \(\text{supp } u\cap \partial G\neq\emptyset\). Here \(P^\sim(\xi)= (\sum_{|\alpha|\leq r} |P^{(\alpha)}(\xi)|^2)^{{1\over 2}}\), \(k\) is a positive weight function, \({\mathcal B}_k= {\mathcal B}_{k, 2}\) are the well-known Sobolev type spaces introduced by Hörmander, and \({\mathcal B}^c_k(G)= {\mathcal B}_k\cap {\mathcal E}'(G)\). This result gives in turn a sufficient condition for \(G\) to be \(P(D)\)-convex, namely \(G\) is \(P(D)\)-convex if for each \(x_0\in \partial G\) there exists a function \(g\) of class \({\mathcal C}^1\) in neighborhood of \(x_0\), such that \(g(x_0)= 0\), \(P_r(\nabla) g(x)\neq 0\) for any \(x\in g^{- 1}(\{0\})\), where \(P_r\) is the principal part of \(P\), and if the set of points where \(g(x)> 0\) is included in \(\mathbb{R}^n\backslash \overline G\) (\(\overline G=\) the closure of \(G\)). The remarkable point of this criterium is this local nature, whether it is well-known that \(P\)-convexity is not. The author gives also another criterium of non \(P\)-convexity in the case \(n= 2\). Finally, he obtains a sufficient condition (of geometric nature) for an open set to be \(P(D)\)-convex. Namely, suppose \(G\) is an open set such that for any \(R> 0\) and any \(x_0\in \partial G\) there exists closed convex sets \(C_1(x_0)\), \(C_2(x_0)\) with the property that \(x_0\in C_2(x_0)\), \(C_1(x_0)\subset C_2(x_0)\subset \mathbb{R}^n\backslash G\) and such that \(C_1(x_0)\subset \mathbb{R}^n\backslash \overline B(0, R)\). Moreover for any characteristic plane \(T\) for \(P\), \(T\cap C_2(x_0)\neq \emptyset\) implies \(T\cap C_1(x_0)\neq \emptyset\). Then \(G\) is \(P(D)\)-convex. A lot of examples illustrates the meaning of the results.