When analyzing agriculture’s total factor productivity, traditional measurement approaches do not take into account the inefficiency value. The production functions are assumed to be analyzed on basis of the random boundaries, which makes the analysis results inaccurate and unreliable. As a result, this paper proposes an analytical approach for agricultural total factor productivity based on the stochastic block model (SBM), which combines the benefits of statistics and machine learning. It uses the SBM direction distance function and the Luenberger productivity index to measure the agricultural efficiency, total factor productivity, and their components. The research study considers the data from 31 provinces from 2006 to 2018 years. First, one output indicator and six input indicators are established. The time-varying variations of the national agricultural inefficiency value and its source decomposition under variable scale returns are then determined using the SBM-based algorithm of agricultural total factor productivity and the obtained sample data. The changes of the agricultural total factor productivity and its components are comprehensively analyzed. Following an examination of the elements impacting agricultural efficiency and productivity, the socioeconomic development of the agricultural total factor productivity is examined in order to achieve efficient growth.