With rapid urbanization in China, the phenomenon of urban sprawl has become prominent and has severely challenged sustainable urbanization and ecological civilization. Aiming to understand the impact of urban sprawl on the urban environment, this study calculates the eco-efficiency of 264 prefecture-level cities in China from 2003 to 2016 by using a super-efficiency data envelopment analysis model. Then, we establish a panel Tobit model and threshold regression model to empirically test the impact of urban sprawl on eco-efficiency and the threshold effect of the urban scale. The results show that urban sprawl hinders the improvement of urban eco-efficiency, especially in Eastern China, but relatively weak or even insignificant effects are observed in Central and Western China. Additionally, a threshold effect of urban sprawl on eco-efficiency can be found. When the city scale is small, urban sprawl seriously hinders the improvement of eco-efficiency. As the city scale gradually expands, the negative effect of urban sprawl on eco-efficiency first decreases, then the restraining effect is gradually strengthened. Our research findings can aid urban development in cities with different scales to reduce the negative effect of urban sprawl on the urban environment.