Construction and demolition activities consume large amounts of natural resources, generating 4.5 bi tons of solid waste/year, called construction and demolition waste (C&DW) and other wastes, such as ceramic, polyethylene terephthalate (PET), glass, and slag. Furthermore, around 32 bi tons of natural aggregate (NA) are extracted annually. In this scenario, replacing NA with recycled aggregate (RA) from C&DW and other wastes can mitigate environmental problems. We review the use of RA for concrete production and draw the main challenges and outlook. RA reduces concrete’s fresh and hardened performance compared to NA, but these reductions are often negligible when the replacement levels are kept up to 30%. Furthermore, we point out efficient strategies to mitigate these performance reductions. Efforts must be spent on improving the efficiency of RA processing and the international standardization of RA.