Featuring complex and fractured terrain, the Loess hilly and gully regions suffer poor grain production capacity. The behavior of farmers, the major users of agricultural production space, significantly influences the agricultural production space. Hence, it is essential to explore the evolution rules of the agricultural production space under the influence of farmer behavior and reveal the influencing mechanism of agricultural production space change, which will facilitate the promotion of ecological protection and high-quality development of the Yellow River Basin. Relying on six-stage remote sensing images of the Yanhe River Basin from 1995 to 2018, this study utilized a land use dynamic index, transfer matrix and landscape pattern index to analyze the spatial pattern evolution of agricultural production in the Yanhe River Basin. Furthermore, the geographic detector model was applied to quantitatively analyze the influencing factors of the spatial pattern evolution of agricultural production. The results demonstrated the following: (1) From 1995 to 2018, the overall area of cultivated land in the Yanhe River Basin decreased by 927.02 km2, with a change degree of 21.07%. The spatial structure of agricultural production changed, mainly transferring the cultivated land to woodland and grassland. (2) The spatial form of agricultural production has changed from fragmentation to regularity, and the complexity of the production space shows a trend of first increasing and then decreasing. (3) The evolution of the spatial pattern of agricultural production was affected by multiple factors of farmer behavior, where significant interactive enhancement effects existed. Specifically, labor input was the dominant factor affecting the overall scale of the production space, with an influence value of 0.202; fertilization input and cultivated land transfer were the key factors affecting the spatial distribution of production, with influence values of 0.264 and 0.242, respectively; income level and social interaction were the base factors affecting the spatial form of production, with influence values of 0.558 and 0.438, respectively. The research results provide scientific support for the improvement of agricultural production quality and the spatial evolution mechanism of agricultural production in the Yanhe River Basin.