In addition to creating economic value, crops also serve important ecological functions. Especially their carbon sink function, which plays a key role in mitigating climate change. In this study, the LMDI and the Tapio model were innovatively combined to quantitatively evaluate the carbon emissions and sinks in the Yellow River basin (YRB). It shows that the average annual growth rate of the YRB was −0.1344% during 2002–2020. Carbon emissions show a negative trend due to the transformation and upgrading of agriculture from traditional to modern and the implementation of policies related to China’s agricultural benefits. Agricultural production efficiency is a major factor in inhibiting agricultural carbon emissions, reducing carbon emissions by an average of approximately 8.07 million tons per year. High-carbon emission and high-sink areas in agriculture are mainly concentrated in the southeast of the YRB, where livestock and poultry farming is the principal source of carbon emissions, with rice, wheat, and corn being the principal contributors to the carbon sink. Moreover, there are significant differences in the carbon sink capacity of crops in the YRB. Optimizing crop selection and area distribution can enhance the carbon sink capacity in different regions, contributing to more effective carbon emission control. This study combines agricultural carbon emissions with the carbon sequestration capacity of crops, providing data support and a theoretical basis for the policy formulation and planning of low-carbon agriculture in China. It is of great significance for promoting sustainable agricultural development and mitigating climate change.