In rural China, the conversion between fine-grained farmland and large-scale farmland is a widespread phenomenon, with different impacts on agricultural carbon emissions. Therefore, how to define the optimal scale of farmland management to achieve " reducing CO2" has become an urgent problem to be solved. Based on the agricultural panel data of Gansu Province for 2000–2020, taking the scale of agricultural land management as the breakthrough point and the consumption intensity of agricultural materials as the intermediary factor, this paper discusses the driving mechanism of agricultural carbon emissions by the scale of agricultural land management. The results including: (1) From the perspective of the intermediary effect, large-scale farmers pay more attention to input efficiency of chemicals such as fertilizers than small-scale farmers, which can effectively promote the development of low-carbon agriculture. (2) A “U-shaped” relationship existed between agricultural land management scale and agricultural carbon emissions. The optimal agricultural land management scale was 7.413 mu/person without considering intermediary variables, and the agricultural carbon emissions were the lowest when agricultural land management scale in Gansu was 9.12 mu/person after adding the intermediary variables. (3) Dividing Gansu into different regions and considering intermediary factors, the carbon emission intensity reached its peak when the scale of farmland management in the Hexi region was 2.14 mu/person. When the optimal scale of farmland management in the Longdong and Longnan regions was 2.14 and 5.22 mu/person, respectively, intermediary variables such as chemical fertilizers in these regions all showed complete intermediary effects.