Due to their potential toxicity and non-degradability, heavy metals pose water and soil quality and safety challenges, impacting crop growth and the ecological environment. The contamination of heavy metals (HMs) and non-point source pollution from agriculture and pastoral presents significant ecological and environmental challenges, necessitating prioritized prevention and mitigation. In this study, 44 water samples and 55 soil samples from Gangba County, a typical agricultural and pastoral area in the Tethys Himalaya tectonic domain, served as research objects. We employed various methods, including the inverse distance weighting, ecological risk assessment model, redundancy analysis, and geographical detector modeling, to investigate the spatial distribution and pollution attributes of arsenic (As), chromium (Cr), cadmium (Cd), lead (Pb), nickel (Ni), nitrogen (N), phosphorus (P), and potassium (K). Our analysis considered the impact of soil physicochemical properties on heavy metals (HMs), elucidating factors influencing their spatial distribution. Results indicated that 65.46% of soil As in the study area exceeded the screening value, while the concentrations of the eight selected elements in water remained below the standard limit. Simultaneously, the study area exhibited low overall ecological risk and minimal HM pollution. Furthermore, As and Pb were primarily linked to human activities and the environment, while Cd, Cr, and Ni were predominantly associated with natural processes. Additionally, factors, such as per capita net income, mean annual temperature, mean annual precipitation, geomorphic type, organic matter, geology type, and soil texture (sand, silt, and clay) constituted primary controlling factors influencing the spatial distribution of HMs in soil. Therefore, for effective prevention and control of HMs and non-point source pollution in agriculture and pastoral, arsenic should be the primary monitoring target, with careful consideration given to the application rates of fertilizers containing N, P, and K to facilitate sustainable development of the ecological environment.