Under the background of global warming, it is of great significance to explore how to realize environmentally sustainable development. This paper takes China’s three major urban agglomerations as the research objects: Yangtze River Delta, Beijing–Tianjin–Hebei, and Pearl River Delta. Generally, we use carbon emission efficiency to represent the sustainable development of the environment. Then we use the city-level panel data of the three urban agglomerations from 2006 to 2019 to construct the slacks-based measure integrating data envelopment (SBM-DEA) model for calculating each city’s carbon dioxide emission efficiency. Finally, we construct the spatial difference-in-differences (SDID) model to explore the impact of high-speed rail construction on each urban agglomeration’s carbon dioxide emission efficiency and its internal mechanism. The findings are as follows: (1) On the whole, high-speed rail construction improves urban agglomerations’ carbon dioxide emission efficiency. Meanwhile, it has a positive spatial spillover effect on surrounding areas. (2) In terms of urban agglomerations, high-speed rail construction has significantly promoted carbon emission efficiency in the Beijing–Tianjin–Hebei region. However, it has had negative external effects on the surrounding areas. (3) From the perspective of mechanism analysis, the construction of high-speed rail has promoted manufacturing agglomeration in the Pearl River Delta region and, at the same time, has had a negative impact on the local carbon dioxide emission efficiency. This study has strong policy implications for promoting the sustainable development of the three major urban agglomerations.