To understand the heavy metal pollution in agricultural soils along the East River basin and assess the pollution related health effect to local residents, interviews and archived data were obtained to identify the study sites affected by polluted tailing. Soil samples were collected and tested for heavy metal content and the Comprehensive Pollution Index (CPI). The degree of pollution of agricultural soils in the area was assessed using GIS-based Spatial distribution map of heavy metals and the trend of soil heavy metal risk. Two villages (Matian and Zhudui) near the East River were included in this study for health effect assessment. A total of 193 residents aged 15 or above from each village were tested for the present status of chronic diseases. Convenient sampling method was used to collect blood samples from 78 residents for heavy metal concentration. The contents of Pb, Cd, As, Zn, and Cu in the agricultural soils were all over the standards with a moderate to severe CPI. Among these metals, Cd was the highest followed by Pb, and Cu was the lowest. The contents of Pb, Cd, As, and Zn tend to be higher in soils closer to the river. The prevalence of chronic diseases was over 30%, which is significantly higher than the report from the national central region (23.15%). The average blood lead level (BLL) among children under 14 years is 7.42 µg/dL. Although the adults in Matian village had a significantly higher BLL (χ 2 = 8.70, p = 0.03) as compared to Zhudui village, there was no significant difference for the prevalence of chronic diseases between the two villages (χ 2 = 3.23, p = 0.09). The mean BLL of children and the proportion of children with BLL ≥ 10 µg/dL in this study are equivalent to the national average. The higher BLL concentration and prevalence of chronic diseases in adults might be due to their long-term exposure to heavy metal contamination environment and higher background level of heavy metals. Findings from this study will form the baseline information for local government to the development of effective approaches to control the heavy metal contamination and reduce the pollution related adverse health effect on local residents.