Shimura curves within the locus of hyperelliptic Jacobians in genus 3 (Q2801961)

A Kuga curve is defined here to be a curve \(\mathcal C\) in \(\mathcal A_g=\mathrm{Sp}(2g,\mathbb Z)\backslash\mathrm{Sp}(2g,\mathbb R)/\mathrm U(g)\) given by a homomorphism \(G\to\mathrm{Sp}(2g,\mathbb R)\) for some algebraic group \(G\). If \(\mathcal C\) contains a CM point, this is what is usually called a Shimura curve, but that condition plays no role in this paper so the authors work with Kuga curves throughout. The aim is to classify, in the case \(g=3\), the Kuga curves that lie in the locus of hyperelliptic Jacobians: more precisely, those whose general point lies in that locus and which are not complete curves in \(\mathcal A_3\). A well-known conjecture asserts that for large \(g\) there should be no Kuga curves (generically) in the Jacobian locus at all: for \(g=3\) this means that any Kuga curve not contained in the locus of decomposible abelian threefolds is already showing exceptional behaviour.NEWLINENEWLINEThe idea is that if one has such a curve one can ask how it meets the toroidal boundary and impose the condition on the period matrices given by the condition that the theta-null modular form vanishes, which is what defines the hyperelliptic locus. Therefore the Fourier-Jacobi expansion vanishes term by term on the curve, and this imposes strong conditions on the period matrices. To do this, we need local coordinates near a boundary point, and these are obtained from the toroidal structure of the compactification, so there are several cases depending on which toric stratum one starts from.NEWLINENEWLINETwo perhaps unexpected things happen here. Firstly, the kind of analysis needed depends on whether the toric stratum is of torus rank 1, 2 or 3. Secondly, and more surprisingly, although the rank 1 and rank 3 cases can be excluded (by different arguments), it turns out that Kuga curves, indeed Shimura curves, do actually exist with torus rank 2 degenerations at the cusps. The possible degeneration data are fully described here, and examples of the Shimura curves are given: the period matrices are of the form NEWLINE\[NEWLINE\begin{pmatrix} t+iu^2 & u^2/2 & iu\\ u^2/2 & t & u\\ iu & u & 1\end{pmatrix}NEWLINE\]NEWLINE for any fixed \(u\in\mathbb Q[i]\setminus\mathbb Z[i]\), and \(t>\min(\Re(u^2),0)\).NEWLINENEWLINEThere is a good description of non-complete Kuga curves, due to Viehweg and Zuo, and the first step is to write this in terms of period matrices. The period matrices are of the form NEWLINE\[NEWLINE\begin{pmatrix} E&0\\ 0 & 0\end{pmatrix} +A\begin{pmatrix} 0 & 0\\ 0 & Z\end{pmatrix} A^T+RNEWLINE\]NEWLINE where \(E\) lies in a toroidal cone \(\sigma\) and \(Z\in\mathbb H_{g-r}\) (where \(r\) is the torus rank) and \(A\in\mathrm{GL}(g,\mathbb Q)\) and \(R\in M_{g\times g}(\mathbb Q)\) are constant.NEWLINENEWLINEIn the case \(g=3\), the authors exclude \(r=1\) by examining the expansion of the theta-null form: there is only one choice of \(\sigma\). (In fact this case is also excluded by a more general theorem of Xiao.) The eventual conclusion is that any such Kuga curve on which the theta-null vanishes must lie in the locus of decomposible abelian \(3\)-folds.NEWLINENEWLINEThe case \(r=3\) can be excluded for similar reasons, but by a different argument: one has a well-defined lowest order term in the Fourier-Jacobi expansion corresponding to each possible cone \(\sigma\), and the condition that this term vanishes can be analysed using a criterion due to \textit{H. B. Mann} [Mathematika 12, 107--117 (1965; Zbl 0138.03102)]. Here the fact that \(g=3\) is being used in two ways: using the theta-null to define the hyperelliptic locus and using the description of the (essentially unique) toroidal compactification.NEWLINENEWLINEThe same method applies for \(r=2\) but this time one choice of \(\sigma\) and the choice of \(E\) such that \(E_{11}=E_{22}\) is not excluded and indeed actually arises, being the examples given above. The rest of paper is devoted to examining this case, showing that there are infinitely many such curves, that they are in fact Shimura curves, and that one of them coincides with a family previously written down by \textit{B. Moonen} and \textit{F. Oort} [in: Handbook of moduli. Volume II. Somerville, MA: International Press; Beijing: Higher Education Press. 549--594 (2013; Zbl 1322.14065)] and studied by \textit{X. Lu} and \textit{K. Zuo} [``On Shimura curves in the Torelli locus of curves'', Preprint, \url{arXiv:1311.5858}].