The combination of general relativity (GR) and the Standard Model of particle physics disagrees with numerous observations on scales from our Solar System up. In the canonical concordance model of Lambda cold dark matter (LCDM) cosmology, many of these contradictions between theory and data are removed or alleviated by the introduction of three completely independent new components of stress energy-the inflaton, dark matter and dark energy. Each of these in its turn is meant to have dominated (or to currently dominate) the dynamics of the Universe. There is, until now, no non-gravitational evidence for any of these dark sectors, nor is there evidence (though there may be motivation) for the required extension of the Standard Model. An alternative is to imagine that it is GR that must be modified to account for some or all of these disagreements. Certain coincidences of scale even suggest that one might expect not to make independent modifications of the theory to replace each of the three dark sectors. Because they must address the most different types of data, attempts to replace dark matter with modified gravity are the most controversial. A phenomenological model (or family of models), modified Newtonian dynamics, has, over the last few years, seen several covariant realizations. We discuss a number of challenges that any model that seeks to replace dark matter with modified gravity must face: the loss of Birkhoff's theorem, and the calculational simplifications it implies; the failure to explain clusters, whether static or interacting, and the consequent need to introduce dark matter of some form, whether hot dark matter neutrinos or dark fields that arise in new sectors of the modified gravity theory; the intrusion of cosmological expansion into the modified force law, which arises precisely because of the coincidence in scale between the centripetal acceleration at which Newtonian gravity fails in galaxies and the cosmic acceleration. We conclude with the observation that, although modified gravity may indeed manage to replace dark matter, it is likely to do so by becoming or at least incorporating a dark matter theory itself.