As typical representatives of China’s industrial sectors, energy intensive industries are the focus of energy conservation. This study constructs a trans-log production function and stochastic frontier analysis model to analyze the impact of energy price distortion on total factor energy efficiency in energy intensive industries on the city level. The results reveal that the phenomenon of energy price distortion existed in all cities from 2003 to 2019, with an average degree of −0.175; and the total factor energy efficiency in China’s energy intensive sectors showed an upward trend, with an average efficiency of 0.729. Further deep analysis of affecting mechanisms concluded that the price distortion showed a significant restraining effect on improving energy efficiency, while the improvement of urban agglomeration systems had an opposite effect. In addition, energy consumption structure, foreign trade, and infrastructure construction are positively correlated with energy efficiency. Therefore, promoting the market-oriented reform of China’s energy market is of great significance to improve city energy efficiency and build a resource-conserving society.