We study the asymptotic safety conjecture for quantum gravity in the presence of matter fields. A general line of reasoning is put forward explaining why gravitons dominate the high-energy behavior, largely independently of the matter fields as long as these remain sufficiently weakly coupled. Our considerations are put to work for gravity coupled to Yang-Mills theories with the help of the functional renormalization group. In an expansion about flat backgrounds, explicit results for beta functions, fixed points, universal exponents, and scaling solutions are given in systematic approximations exploiting running propagators, vertices, and background couplings. Invariably, we find that the gauge coupling becomes asymptotically free while the gravitational sector becomes asymptotically safe. The dependence on matter field multiplicities is weak. We also explain how the scheme dependence, which is more pronounced, can be handled without changing the physics. Our findings offer a new interpretation of many earlier results, which is explained in detail. The results generalize to theories with minimally coupled scalar and fermionic matter. Some implications for the ultraviolet closure of the Standard Model or its extensions are given.