There are differences in resource endowments, environmental regulations, and production functional areas among different regions in China, which makes dairy farming have certain heterogeneity and imbalance in pollution control, which is mainly reflected in the differences in the environmental efficiency of dairy farming among regions. By analyzing the convergence trend of the dairy farming environment in different regions and scales, it is helpful to predict the potential of improving environmental efficiency, and promote the high-quality development of dairy farming. Based on the dairy farming data of 27 provinces (cities and districts) in China from 2004 to 2019, the SBM model of unexpected output was used to measure the environmental efficiency of dairy farming on different scales. It was found that there were certain differences in the environmental efficiency of dairy farming in China. The overall environmental efficiency of dairy farming showed a “U" evolution trend, including small-scale, medium-scale, and large-scale. The average values of their environmental efficiency were 0.6859, 0.6930, and 0.6855 respectively, and the environmental efficiency of medium-scale dairy farming was the best. The convergence analysis of environmental efficiency showed that there was a conflict between the traditional σ convergence and β convergence results. However, the club convergence showed that the environmental efficiency of small-scale dairy farming converged in general, which could be further divided into two club convergence. The environmental efficiency of medium-scale and large-scale dairy farming did not converge in general but did converge in four different clubs, respectively. Taking the highest environmental efficiency value in the “club” as the reference, the potential to improve the environmental efficiency of dairy farming in different sizes was calculated. The prediction results showed that the gap between the actual value and the potential value was narrowing. It is expected that the potential to improve the environmental efficiency of dairy farming in China will remain above 10% in the next 10–20 years. Based on the above studies, this paper put forward some policy suggestions, such as changing the development direction of dairy farming, changing scale expansion to structural upgrading, implementing differentiated environmental efficiency improvement strategies, narrowing the efficiency gap between regions, paying attention to the input-output structure of factors, and improving the efficiency of breeding environment from the perspective of “eating” and " emission ".