Recent observations of isotropic diffuse backgrounds by Fermi and IceCube allow us to get more insight into distant very-high-energy (VHE) and ultra-high-energy (UHE) gamma-ray/neutrino emitters, including cosmic-ray accelerators/sources. First, we investigate the contribution of intergalactic cascades induced by gamma-rays and/or cosmic rays (CRs) to the diffuse gamma-ray background (DGB) in view of the latest Fermi data. We identify a possible "VHE Excess" from the fact that the Fermi data are well above expectations for an attenuated power law, and show that cascades induced by VHE gamma rays (above ∼ 10 TeV) and/or VHECRs (below ∼ 10 19 eV) may significantly contribute to the DGB above ∼ 100 GeV. The relevance of the cascades is also motivated by the intergalactic cascade interpretations of extreme TeV blazars such as 1ES 0229+200, which suggest very hard intrinsic spectra. This strengthens the importance of future detailed VHE DGB measurements. Then, more conservatively, we derive general constraints on the cosmic energy budget of high-energy gamma rays and neutrinos based on recent Fermi and IceCube observations of extragalactic background radiation. We demonstrate that these multi-messenger constraints are useful and the neutrino limit is very powerful for VHE/UHE hadronic sources. Furthermore, we show the importance of constraints from individual source surveys by future imaging atmospheric Cherenkov telescopes such as Cherenkov Telescope Array, and demonstrate that the cascade hypothesis for the VHE DGB can be tested by searching for distant emitters of cascaded gamma rays.