A stochastic frontier approach (SFA) model of translog production function was constructed to analyze the growth effect of agricultural production factors on grain production in China. Under the condition of unchanged cultivated land, the agricultural labor, capital, and water were regarded as input elements of the agricultural production function. The maximum likelihood estimation (MLE) method was used to analyze the technical efficiency, output elasticity, substitution elasticity, and relative variability of grain production in China from 2004 to 2018. The results showed that: (1) For the technical efficiency and output elasticity of the input factors of grain production, there were significant differences in different provinces. For example, the water resource was insufficient in Beijing and Shanghai, but the output elasticity of water was high. Heilongjiang was rich in water and had high technical efficiency. For Xinjiang, water was sufficient, but its output elasticity was deficient and the technical efficiency didn’t increase. (2) The overall technical efficiency level was relatively low and was still declining year by year; the output elasticity of water was much greater than that of capital. There was still great potential for grain growth. (3) Optimizing resource allocation and controlling the appropriate ratio of input factors to develop grain production could achieve the maximum benefits. Finally, according to the empirical results, this paper put forward some practical policy suggestions for optimizing the allocation of input factors, especially water and capital, which can ultimately improve agricultural productivity by improving technical efficiency.