Food security is an important issue affecting people’s lives and social stability. Clarifying levels of food security and the factors affecting it (social, economic, agricultural, climatic) can help improve regional food security. The spatiotemporal patterns and driving factors of food security vary at different scales. There is, however, a lack of research that considers the various factors affecting food security at multiple scales. This study, therefore, analyzed dynamic spatiotemporal changes in food security at small (city), medium (province), and large (country) scales; identified hot and cold areas of food security; and revealed the main factors affecting food security at different scales. A food security index (FSI) was built based on the coupling of grain yield, population, and GDP, and spatial analysis was used to evaluate dynamic spatiotemporal changes in China’s food security from 1980 to 2017. Further, the relationship between food security and its driving factors was quantitatively analyzed using stepwise regression. The results showed greater heterogeneity in food security at the smaller scale than at the larger scale. The key factors affecting food security varied substantially at different scales: the added value of tertiary industry dominated the prefecture level, and gross agricultural output value was the main factor at the provincial and national levels. Multiple-scale research can reveal the status and primary factors of food security and provide a decision-making basis for improving regional food security.