On trinomials of type \(X^{n+m}(1+AX^{m(q -1)}+BX^{n(q -1)})\), \(n,m\) odd, over \(\mathbb{F}_{q^2}\), \(q=2^{2s+1}\) (Q2662049)

Let \(q=2^{2s+1}\), where \(s\in\mathbb N\), and let \[ F_{A,B,m,n}(X)=X^{m+n}(1+AX^{m(q-1)}+BX^{n(q-1)})\in\mathbb F_{q^2}[X], \] where \(m\) and \(n\) are positive odd integers such that \(\text{gcd}(m,q+1)=\text{gcd}(m,n)=1\), \(A\in\mathbb F_q^*\) and \(B\in\mathbb F_{q^2}^*\). The aim of the paper is to determine necessary conditions on the parameters of \(F_{A,B,m,n}\) such that it is a permutation polynomial (PP) of \(\mathbb F_{q^2}\), i.e., \(F_{A,B,m,n}\) induces a bijection from \(\mathbb F_{q^2}\) to \(\mathbb F_{q^2}\). The polynomial \(F_{A,B,m,n}\) belongs to a more general class \(X^rh(X^{q-1})\), where \(h\in\mathbb F_{q^2}[X]\); PPs in this class have been extensively studied. It is well known that \(F_{A,B,m,n}\) permutes \(\mathbb F_{q^2}\) if and only if \(X^{m+n}(1+AX^m+BX^n)^{q-1}\) permutes \(\mu_{q+1}:=\{x\in\mathbb F_{q^2}^*:x^{q+1}=1\}\). The latter condition can be restated as a certain plane curve \(\overline{\mathcal E}_{A,B,m,n}\) having no \(\mathbb F_q\)-rational points. If \(\overline{\mathcal E}_{A,B,m,n}\) has no \(\mathbb F_q\)-rational points, then the Hasse-Weil bound implies that \(\overline{\mathcal E}_{A,B,m,n}\) has no absolutely irreducible factors over \(\mathbb F_q\) when its degree is \(<q^{1/4}\). Assume that \(\overline{\mathcal E}_{A,B,m,n}\) has no absolutely irreducible factors over \(\mathbb F_q\). It is shown that \(\overline{\mathcal E}_{A,B,m,n}\) is defined by \(\mathcal A(X,Y)\mathcal B(X,Y)=0\), where \(\mathcal A, \mathcal B\in\mathbb \overline{\mathbb F_q}[X,Y]\), \(\text{gcd}(\mathcal A,\mathcal B)=1\) and \(\deg\mathcal A\cdot\deg\mathcal B\ge 8(m+n)^2/9\). Bézout's theorem dictates that \[ \frac{8(m+n)^2}9\le \deg\mathcal A\cdot\deg\mathcal B =\sum_{P\in\mathcal A\cap\mathcal B}I(P,\mathcal A\cap\mathcal B), \] where \(I(P,\mathcal A\cap\mathcal B)\) is the intersection multiplicity of \(\mathcal A\) and \(\mathcal B\) at \(P\). An upper bound for the sum of intersection multiplicities is obtained by considering another plane curve \(\overline{\mathcal D}_{A,B,m,n}\) which is projectively equivalent to \(\overline{\mathcal E}_{A,B,m,n}\) and by analyzing the singular points on \(\overline{\mathcal D}_{A,B,m,n}\). (This part is quite technical.) The main theorem follows from a comparison of the lower and upper bounds for the sum of intersection multiplicities: \medskip Theorem 6.1. If \(F_{A,B,m,n}(X)\) is a PP of \(\mathbb F_{q^2}\), then one of the following conditions is satisfied, where \(n\equiv i\pmod 8\) and \(m\equiv j\pmod 8\). \begin{itemize} \item[(1)] \(2(n+m)\ge q^{1/4}\). \item[(2)] \(B\in\mathbb F_q\). \item[(3)] \(A^2+B^{q+1}\ne 0\), \(B\notin\mathbb F_q\), and \(n<5m\). \item[(4)] \(A^2+B^{q+1}=0\), \(B\in\mathbb F_q\), \((i,j)\in\{(1,5),(3,7),(5,1),(7,3)\}\), and \(n<38 m\). \item[(5)] \(A^2+B^{q+1}=0\), \(B\notin\mathbb F_q\), \((i,j)\notin\{(1,5),(3,7),(5,1),(7,3)\}\). \end{itemize}