Heavy metal pollution related to non-ferrous metal smelting may pose a significant threat to human health. This study analyzed 58 surface soils collected from a representative non-ferrous metal smelting area to screen potentially hazardous heavy metals and evaluate their health risk in the studied area. The findings demonstrated that human activity had contributed to the pollution degrees of Cu, Cd, As, Zn, and Pb in the surrounding area of a non-ferrous metal smelting plant (NMSP). Cu, Cd, As, Zn, Pb, Ni, and Co pollution within the NMSP was serious. Combining the spatial distribution and Spearman correlations with principal component analysis (PCA), the primary sources of Cd, As, Pb, and Zn in surrounding areas were related to non-ferrous metal smelting and transportation activities. High non-cancer (THI = 4.76) and cancer risks (TCR = 2.99 × 10−4) were found for adults in the NMSP. Moreover, heavy metals in the surrounding areas posed a potential cancer risk to children (TCR = 3.62 × 10−6) and adults (TCR = 1.27 × 10−5). The significant contributions of As, Pb, and Cd to health risks requires special attention. The construction of a heavy metal pollution management system will benefit from the current study for the non-ferrous metal smelting industry.