Green innovation possesses dual externalities of “innovation” and “environmental protection”, and enhancing energy efficiency serves as a crucial means to promote high-quality economic development. Building upon the energy rebound effect, we use the balanced panel data of cities at prefecture level and above in China from 2008 to 2018 to explore the impact of urban green innovation on total factor energy efficiency (TFEE). The findings of this study indicate that, firstly, the impact of green innovation on TFEE exhibits a positive U-shaped pattern, characterized by initial suppression followed by subsequent promotion. This conclusion remains robust after undergoing a series of rigorous robustness tests. Second, high-quality green innovation is found to reach the turning point more quickly, implying that substantial green innovation can cross the turning point in smaller quantities. Thirdly, on the whole, in comparison to non-resource-based cities, resource-based cities are capable of reaching the turning point at an earlier stage. Finally, the new energy demonstration cities have not yet reached the turning point, while the non-new energy demonstration cities have crossed the turning point. This study contributes novel insights into the relationship between green innovation and TFEE, which holds significant implications for the formulation and implementation of sustainable development policies.