Soil pollution by metal(loid)s caused by smelting activities is a severe problem posing a great threat to environmental and human health. In this study, the concentrations, sources and human health risks posed by six potentially toxic elements (Cr, Mn, Zn, Pb, Cd, and As) were determined in the soil of a typical alloy smelting site in South Central China. The results showed that the concentrations of metal(loid)s were in the descending order of Mn > Cr > Zn > Pb > As > Cd. Additionally, the selected elements were classified into different pollution degrees using geo-accumulation index and pollution load index. The entire study area had a high pollution level with relatively severe and extensive contamination by Cr, Cd, and Mn. The combined application of principal component analysis and positive matrix decomposition model revealed that the major sources of these elements include smelting activities (48.68%), waste residue stacking (22.95%), and natural sources (28.37%). According to the results of the human health risk assessment, the non-carcinogenic risk was insignificant as a whole; however, the carcinogenic risk had an unacceptable level. Among them, Cr was the main driver of carcinogenic risk, which needs special attention.