Based on interviews with 695 smallholders in Jianghan Plain, this paper introduced the three-stage data envelopment analysis (DEA) model to analyze the agricultural production efficiency of conservation tillage adopters and explored the impact of environmental factors on agricultural production efficiency. The empirical results showed the following (1) Planting area, seed consumption, labor input, pesticide usage, chemical fertilizer usage, agricultural film usage were selected as input indicators, agricultural output was chosen as an output indicator, and the traditional DEA model was used to calculate the production efficiency of smallholders, and the agricultural production efficiency of smallholders was found to be at a low level. In addition, environmental and random factors both have significant impacts on efficiency, so they should be stripped. (2) After excluding environmental factors and random factors, the drop in pure technical efficiency of smallholders in the third stage was higher than the drop in scale efficiency when compared with the first stage. Moreover, the true technical efficiency was the main restricting factor for the agricultural production efficiency. (3) Educational level of smallholders, policy support, and information acquisition were the factors that affect the technical efficiency significantly. Improving the efficiency of agricultural production technology for smallholders requires strengthening rural basic education, improving subsidy policies for conservation agricultural technology, and establishing and improving rural information technology services.