Highlighting the nexus between poverty and environment is essential to promote poverty alleviation and sustainable development. This study analyzed spatial differences and environmental factors influencing poverty incidence and reduction using spatial statistical methods and GeoDetector tools. It focused on Lijiang in the Hengduan Mountains of western China as the case area. The results indicate a notable decline in poverty incidence in most Lijiang villages during 2014–2018 under China’s poverty alleviation strategy. However, there are distinct spatial differences for both poverty incidence and reduction. The main environmental factors affecting poverty incidence and reduction are available water storage and geological hazard risks. Socioeconomic factors such as administrative unit and distance to city center also play a key role. The anti-poverty policies in various administrative units have the most significant influence. However, existing policy formulation mainly considers elevation factor in mountainous areas. This study suggests that water resources and geological hazards should also be highly considered, and not only elevation. It is imperative to promote the construction of water conservancy facilities and improve the prevention and control of geological disasters. Moreover, targeted poverty alleviation should focus on, not only policy or socioeconomic factors, but also main environmental factors affecting poverty incidence and reduction.