In low mountain and hilly regions, vegetation cover is higher and plant growth has an accumulative effect, sequestering carbon more strongly. The traditional remote sensing based ecological index (RSEI) lacks the consideration of vegetation productivity, and using it to evaluate ecological environment in low mountain and hilly regions will be biased. In this study, the vegetation productivity was introduced to construct a natural remote sensing based ecological index (NRSEI) that responds to the low mountain and hilly regions, as an example of Gaizhou City, China. Additionally, this study explored the spatiotemporal evolution of ecological environment quality from 2014 to 2020 and quantified the influences of factors. The results show that the first principal component (PC1) increased from 56 to 67% to 65–87% and considered the accumulation process in the ecosystem. NRSEI was more valid. From 2014 to 2020, the quality of the ecological environment generally declined and then increased. The area with “Excellent” increased from 23 to 38%. The quality of ecosystems in the west, northwest, and south deteriorated significantly, a distribution pattern of “high in the center, low in the north and south”. Landuse and topographic conditions dominate the impacts on the ecosystem in the context of social, economic and policy influences. The interactions of the factors were two-factor enhancement that together affect the ecological environment. The results contribute to the development of urban conservation policies in low mountain and hilly regions.