Pyrochlore systems are ideally suited to the exploration of geometrical frustration in three dimensions, and their rich phenomenology encompasses topological order and fractional excitations. Classical spin ices provide the first context in which it is possible to control emergent magnetic monopoles, and anisotropic exchange leads to even richer behaviour associated with large quantum fluctuations. Whether the magnetic ground state of Yb2Ti2O7 is a quantum spin liquid or a ferromagnetic phase induced by a Higgs transition appears to be sample dependent. Here we have determined the role of structural defects on the magnetic ground state via the diffuse scattering of neutrons. We find that oxygen vacancies stabilise the spin liquid phase and the stuffing of Ti sites by Yb suppresses it. Samples in which the oxygen vacancies have been eliminated by annealing in oxygen exhibit a transition to a ferromagnetic phase, and this is the true magnetic ground state.