Metals in soils exert toxic effects on soil quality and human health. In order to carry out metal contamination assessment and human health risk assessment of agricultural soils, a total of 662 cultivated soil samples were collected from two industrially oriented towns and three agriculturally oriented towns of Shifang City, Southwest China. Various indices were applied to analyze the pollution levels of soils caused by metals (Cr, Fe, Pb, Sr, Ti, V, Zn, and Zr). The relatively high values of the geo‐accumulation index (Igeo) and the enrichment factor (EF) suggested that Sr, V, and Zn had cumulative trends in the soils. The average pollution load index (PLI) values of the industrial towns (1.19 and 0.99) were higher than those of the agricultural towns (0.92, 0.53, and 0.43), demonstrating the contribution of industrial activities to soil metal enrichment. The health risk assessment from the five towns indicated that Cr and V were the dominant contaminants in soils with high potential health risks. Dermal contact was the major pathway of the exposure of children and adults to Cr and V. This study improved the understanding of the pollution levels of metals in Shifang agricultural soils and put forward the necessity of minimizing the health risk of Cr and V in agricultural soil.