The extragalactic background light (EBL) observed at multiple wavelengths is a promising tool to probe the nature of dark matter. This radiation might contain a significant contribution from gamma-rays produced promptly by dark matter particle annihilation in the many halos and subhalos within our past-light cone. Additionally, the electrons and positrons produced in the annihilation give energy to the cosmic microwave photons to populate the EBL with X-rays and gamma-rays.To study these signals, we create full-sky maps of the expected radiation from both of these contributions using the high-resolution Millennium-II simulation of cosmic structure formation. Our method also accounts for a possible enhancement of the annihilation rate by a Sommerfeld mechanism due to a Yukawa interaction between the dark matter particles prior to annihilation. We use upper limits on the contributions of unknown sources to the EBL to constrain the intrinsic properties of dark matter using a model-independent approach that can be employed as a template to test different particle physics models. These upper limits are based on observational measurements spanning eight orders of magnitude in energy (from soft X-rays measured by the CHANDRA satellite to gamma-rays measured by the Fermi satellite), and on expectations for the contributions from non-blazar active galactic nuclei, blazars and star forming galaxies. To exemplify this approach, we analyze a set of benchmark Sommerfeld-enhanced models that give the correct abundance of dark matter, satisfy constraints from the cosmic microwave background, and fit the cosmic ray spectra measured by PAMELA and Fermi without any contribution from local substructure. We find that these models are in conflict with the EBL constraints unless the contribution of unresolved substructure is small and the dark matter annihilation signal dominates the EBL. We conclude that provided the collisionless cold dark matter paradigm is accurate, even for conservative estimates of the contribution from unresolved substructure and astrophysical backgrounds, the EBL is at least as sensitive a probe of these types of scenarios as the cosmic microwave background. More generally, our results disfavor an explanation of the positron excess measured by the PAMELA satellite based only on dark matter annihilation in the smooth Galactic dark matter halo.PACS numbers: 95.35.+d,95.85.Nv,95.85.Pw