Computing nonsurjective primes associated to Galois representations of genus $2$ curves

Abstract

For a genus 2 curve C over Q whose Jacobian A admits only trivial geometric endomorphisms, Serre's open image theorem for abelian surfaces asserts that there are only finitely many primes for which the Galois action on -torsion points of A is not maximal. Building on work of Dieulefait, we give a practical algorithm to compute this finite set. The key inputs are Mitchell's classification of maximal subgroups of PSp 4 (F ), sampling of the characteristic polynomials of Frobenius, and the Khare-Wintenberger modularity theorem. The algorithm has been submitted for integration into Sage, executed on all of the genus 2 curves with trivial endomorphism ring in the LMFDB, and the results incorporated into the homepage of each such curve on a publicly-accessible branch of the LMFDB.