Unreasonable human activities may cause the accumulation of heavy metals (HMs) in the agricultural soil, which will ultimately threaten the quality of soil environment, the safety of agricultural products, and human health. Therefore, the accumulation characteristics, potential sources, and health risks of HMs in agricultural soils in China’s subtropical regions were investigated. The mean Hg, Cu, Zn, Pb, and Cd concentrations of agricultural soil in Jinhua City have exceeded the corresponding background values of Zhejiang Province, while the mean concentrations of determined 8 HMs were less than their corresponding risk-screening values for soil contamination of agricultural land in China. The spatial distribution of As, Cr, Ni, Cu, and Pb were generally distributed in large patches, and Hg, Zn, and Cd were generally sporadically distributed. A positive definite matrix factor analysis (PMF) model had better performance than an absolute principal component–multiple linear regression (APCS-MLR) model in the identification of major sources of soil HMs, as it revealed higher R2 value (0.81–0.99) and lower prediction error (−0.93–0.25%). The noncarcinogenic risks (HI) of the 8 HMs to adults and children were within the acceptable range, while the carcinogenic risk (RI) of children has exceeded the safety threshold, which needs to be addressed by relevant departments. The PMF based human health risk assessment model indicated that industrial sources contributed the highest risk to HI (32.92% and 30.47%) and RI (60.74% and 61.5%) for adults and children, followed by agricultural sources (21.34%, 29.31% and 32.94% 33.19%). Therefore, integrated environmental management should be implemented to control and reduce the accumulation of soil HMs from agricultural and industrial sources.