Single Hurwitz numbers enumerate branched covers of the Riemann sphere with specified genus, prescribed ramification over infinity, and simple branching elsewhere. They exhibit a remarkably rich structure. In particular, they arise as intersection numbers on moduli spaces of curves and are governed by the topological recursion of Chekhov, Eynard and Orantin. Double Hurwitz numbers are defined analogously, but with prescribed ramification over both zero and infinity. Goulden, Jackson and Vakil have conjectured that double Hurwitz numbers also arise as intersection numbers on moduli spaces.In this paper, we repackage double Hurwitz numbers as enumerations of branched covers weighted by certain monomials and conjecture that they are governed by the topological recursion. Evidence is provided in the form of the associated quantum curve and low genus calculations. We furthermore reduce the conjecture to a weaker one, concerning a certain polynomial structure of double Hurwitz numbers. Via the topological recursion framework, our main conjecture should lead to a direct connection to enumerative geometry, thus shedding light on the aforementioned conjecture of Goulden, Jackson and Vakil.