In recent years, China’s agriculture-oriented characteristic towns (AOCTs) have developed rapidly. However, the development of AOCTs in different regions varies. Having examined 131 AOCTs in China, this paper constructs a comprehensive evaluation system for the sustainable development of AOCTs and applies a sub-constraint evaluation model to study the performance of AOCTs. The main findings are as follows. (1) The most important dimension for the sustainable development of AOCTs is “social development”, which implies that in order to build AOCTs, the government should pay attention to social development in rural areas, especially improving medical services and public service. (2) Under the dimension of “social development”, the three most important factors are the unemployment rate, per capita consumption expenditures of rural households, and per capita consumption expenditures of urban households. This implies that we must adhere to the policy for ensuring rural employment and encourage the development of local enterprises in order to provide more jobs and change their existing consumption structure. (3) Among the four main economic zones in China, the eastern region has the most advantageous factors, and the western region has the least; furthermore, regarding the disadvantaged factors, the western region has the most and the eastern region the least. This shows that AOCTs in different regions should determine their respective advantages and disadvantages, match resources accordingly, and formulate their own development strategy, which could also contribute to decreasing the gap between the eastern and western regions. Hence, the results of this study not only clearly point out the important factors for the sustainable development of AOCTs but also make them detailed and specific in order to provide the government with targeted and highly operable suggestions.