We prove that the ideal of the variety of secant lines to a Segre-Veronese variety is generated in degree three by minors of flattenings. In the special case of a Segre variety this was conjectured by Garcia, Stillman and Sturmfels, inspired by work on algebraic statistics, as well as by Pachter and Sturmfels, inspired by work on phylogenetic inference. In addition, we describe the decomposition of the coordinate ring of the secant line variety of a Segre-Veronese variety into a sum of irreducible representations under the natural action of a product of general linear groups.