The reasonable allocation of grain production factors is of vital importance to food security and agricultural development. To assess the impact of agricultural factor misallocation on food security, this paper, based on the panel data from China spanning from 2005 to 2019, conducted a comprehensive evaluation of agricultural factor misallocation and food security coefficients across 31 provinces in China, using a spatial Durbin model to examine the effects of factor misallocations on food security. The findings are as follows: (1) Production factor misallocation has significant negative impacts on food security, among them, capital misallocation and labor misallocation inhibiting food security in the local and neighboring areas, and land misallocation has a significant negative impact on food security in local areas, while its spatial spillover effect is no longer significant. (2) Mechanism analysis shows that capital misallocation and labor misallocation hinder the development of transportation infrastructure and the transfer of rural labor, thereby reducing food security. Land misallocation has accelerated the construction of transportation infrastructure, promoted the migration of rural labor, and helped ensure food security. (3) The regional heterogeneity test reveals that capital misallocation and labor misallocation hinder food security in major grain-producing areas and both sides of the Hu Huanyong Line (Hu Line). Meanwhile, land misallocation hinders food security development in various grain-producing areas, as well as the southeast of the Hu Line. Based on the above conclusions, this paper proposes suggestions to improve the efficiency of land and labor resource allocation, accelerate the construction of transportation infrastructure, and encourage the transfer speed of surplus agricultural labor.