The initial energy transfer steps in photosynthesis occur on ultrafast timescales. We analyze the carotenoid to bacteriochlorophyll energy transfer in LH2 Marichromatium purpuratum as well as in an artificial light-harvesting dyad system by using transient grating and two-dimensional electronic spectroscopy with 10 fs time resolution. We find that Förster-type models reproduce the experimentally observed 60 fs transfer times, but overestimate coupling constants, which lead to a disagreement with both linear absorption and electronic 2D-spectra. We show that a vibronic model, which treats carotenoid vibrations on both electronic ground and excited states as part of the system's Hamiltonian, reproduces all measured quantities. Importantly, the vibronic model presented here can explain the fast energy transfer rates with only moderate coupling constants, which are in agreement with structure based calculations. Counterintuitively, the vibrational levels on the carotenoid electronic ground state play the central role in the excited state population transfer to bacteriochlorophyll; resonance between the donor-acceptor energy gap and the vibrational ground state energies is the physical basis of the ultrafast energy transfer rates in these systems.