Mountain building results in high erosion rates and the interaction of rocks with the atmosphere, water and life. Carbon transfers that result from increased erosion could control the evolution of Earth's long-term climate. For decades, attention has focused on the hypothesised role of mountain building in drawing down atmospheric carbon dioxide (CO2) via silicate weathering. However, it is now recognized that mountain building and erosion affect the carbon cycle in other important ways. For example, erosion mobilises organic carbon (OC) from terrestrial vegetation, transferring it to rivers and sediments and thereby acting to draw down atmospheric CO2 in tandem with silicate weathering. Meanwhile, exhumation of sedimentary rocks can release CO2 through the oxidation of rock OC and sulfide minerals. In this Review we examine the mechanisms of carbon exchange between rocks and the atmosphere and discuss the balance of CO2 sources and sinks. Our Review demonstrates that OC burial and oxidative weathering, not widely considered in most models, control the net CO2 budget associated with erosion. Therefore, lithology strongly influences the impact of mountain building on the global carbon cycle, with an orogeny dominated by sedimentary rocks, and thus abundant rock OC and sulfides, tending towards being a CO2 source.