Analytic hypoellipticity, representations of nilpotent groups, and a nonlinear eigenvalue problem (Q1320636)

The aim of this article is to give new examples of hypoelliptic operators which are not analytic hypoelliptic. Since the example of Baouendi- Goulaouic \(D^ 2_ t + t^ 2D^ 2_ x + D^ 2_ y\), many other examples were given by G. Métivier, J. Sjöstrand and others. As typical example, the author gets here that, for a generic homogeneous polynomial \(P\) of positive degree, in two non-commuting variables, the operator \({\mathcal L} = P(\partial_ x, \partial_ y - X^{m-1} \partial_ t)\) is for \(m \geq 3\) not analytic hypoelliptic in any neighborhood of the origin in \(\mathbb{R}^ 3\). This generalizes previous results obtained by B. Helffer, Pham the Lai-Robert and the author. All these examples are connected with the study of left invariant homogeneous operators on stratified nilpotent groups.