According to the study of the environmental impact of green agricultural development, this paper selects 12 basic indicators from the three dimensions of “ecological agriculture, green production, and output benefits” for the construction of China’s green agricultural development index. There are eight basic indicators used in the construction of the environmental index system, including the area of nature reserves and the level of environmental protection expenditures. It is based on the panel data of 13 provinces in China from 2011 to 2019. For the calculation and analysis of the green agricultural development index and the environmental index, the entropy weight method is used. The calculation results show that there are significant differences in the development level of green agriculture and environmental development among these 13 major provinces; the green agriculture level and environmental development level of Jiangsu Province and Sichuan Province are in the leading position; further using dynamic panel regression, the results show that the development of green agriculture has a significant role in promoting the improvement of the environment, and in the early stage of green agriculture development, the effect of improving the environment is more obvious; the index of green agriculture development can be further divided into ecological agriculture index, green production index and output benefit index. As a result, the improvement of ecological agriculture, green production, and output benefits can contribute to the optimization of the ecological environment. As well, the development of green agriculture has some intermediary effects on the optimization of the environment. It is the intermediary effect of promoting environmental optimization through guiding residents’ green consumption that has the greatest effect, while the intermediary effect of promoting industrial structure upgrading and enterprise technology innovation has a relatively small effect. The empirical results presented in this paper suggest that a variety of measures can be taken to improve the environment, including improving green agriculture development in various regions, increasing government expenditures, and promoting the transformation of economic structures in order to improve the environment.