To improve energy-use sustainability in cities, we proposed a set of urban energy performance assessment indicators and influencing factors based on existing theory and literature. An urban energy performance assessment and influencing factor model was also constructed by the improved stochastic frontier analysis method, and panel data from provincial capitals in China from 2004 to 2013 were considered as an example to carry out an empirical study. Chosen from both endogenous and exogenous perspectives, the urban energy performance assessment indicators and influencing factors take into consideration the capital, labor, energy, urban economic output, urbanization level, population, area, urban climate, and travel selection. Because it considers both random errors and the inefficiency levels of urban productions, the urban energy performance assessment and influencing factor model could reduce the errors caused by two-stage performance assessment and factor analysis, quantify the effects of assessment indicators and influencing factors on urban energy performance, and reflect the actual performance of different cities. Empirical results show that the urban energy performance of provincial capitals in China has been increasing. Chinese provincial capitals also have great potential for energy saving. It was necessary to include energy input as an assessment indicator when evaluating urban energy performance. Population density and urban energy performance showed a negative correlation, but the urbanization rate, temperature index, and household car ownership were positively related to urban energy performance. The urban energy performance of Chinese provincial capitals gradually decreased from east to west. Based on these results, several policy suggestions on urban energy performance development are proposed.