With the expansion of the scale of China’s economy and the acceleration of urbanization, energy consumption is increasing, and environmental degradation and other problems have arisen. In order to solve such prominent problems, China proposed the “carbon peak” and “carbon neutral” targets in 2020. Although there are research conclusions about the impact of urbanization on energy intensity (EI), conclusions about the impact of the urban agglomeration policy (UAP) on EI are still unclear. Therefore, the article studies the impact of the urban agglomeration policy on EI in 279 prefecture-level cities by constructing a Difference-In-Differences (DID) model and mediating effect model. The results show that UAP has a significant effect on reducing EI, but their effects are different with the impact of urban heterogeneity, and the urban agglomeration policy of “Core” cities is less effective than those of “Edge” cities. From the perspective of the influencing mechanism, UAP takes green innovation capability as the intermediary variable to influence EI. The placebo test, PSM-DID regression, counterfactual test, and instrumental variable method all reflect the robustness of the research conclusions. Based on this, the paper puts forward some suggestions for urban agglomeration planning and green technology innovation.