The contamination of heavy metals (HMs) in the topsoil of agricultural and pastoral areas threatens the yield and quality of agriculture and animal husbandry and also endangers regional ecological security and human health. This study analyzed the HMs (Cd, Pb, As, Cu, Zn, Hg, and Mn) in areas with intensive agricultural and pastoral activities to identify their accumulation, source apportionment, and ecological-health risks using geochemical methods coupled with positive matrix factorization (PMF). It was uncovered that the concentrations of HMs presented varying scales of enrichment and contamination levels in the topsoil. However, except for only one sample, the residual HMs concentrations fell within the associated risk screening values (the RSVs). The combined assessment of geo-accumulation index (Igeo) and enrichment factor (EF) pointed that Hg and Cd were the primary contaminants. The PMF identified four sources of HMs including natural geogenic, industry mining, composite (caused by multi-anthropogenic activities), and agricultural sources accounting for 33.7%, 41.0%, 7.7%, and 17.6% of contribution rates, respectively. The potential ecological risk was largely attributed to the accumulation of Hg, followed by Cd. The non-carcinogenic risk for both groups fell beneath the acceptable risk threshold, and the prospective carcinogenic health threats posed by Cd, As, and Pb for different groups should not be ignored, particularly concerning children. It is of great significance to control the HMs pollution and restore soil cleanliness in the study area.