Asbestos is widely used in construction, manufacturing, and other common industrial fields. Human activities such as mining, processing, and transportation can release heavy metals from asbestos into the surrounding soil environment, posing a health hazard to the mining area's environment and its surrounding residents. The purpose of the present study was to determine the extent of ecological and human health damage caused by asbestos pollution, as well as the primary contributors to the contamination, by examining a large asbestos mine and the surrounding soil in China. The level of heavy metal pollution in soil and sources were analyzed using methods such as the geo-accumulation index (Igeo), potential ecological risk index (RI), and positive matrix factorization (PMF) model. A Monte Carlo simulation-based health risk model was employed to assess the health risks of heavy metals in the study area’s soil to human beings. The results showed that the concentrations of As, Pb, Cr, Cu, and Ni in the soil were 1.74, 0.13, 13.31, 0.33, and 33.37 times higher than the local soil background values, respectively. The Igeo assessment indicated significant accumulation effects for Ni, Cr, and As. The RI evaluation revealed extremely high comprehensive ecological risks (RI ≥ 444) in the vicinity of the waste residue heap and beneficiation area, with Ni exhibiting strong individual potential ecological risk (Eir ≥ 320). The soil health risk assessment demonstrated that As and Cr posed carcinogenic risks to adults, with mean carcinogenic indices (CR) of 1.56E − 05 and 4.14E − 06, respectively. As, Cr, and Cd posed carcinogenic risks to children, with mean CRs of 1.08E − 04, 1.61E − 05, and 2.68E − 06, respectively. Cr also posed certain non-carcinogenic risks to both adults and children. The PMF model identified asbestos contamination as the primary source of heavy metals in the soil surrounding the asbestos mining area, contributing to 79.0%. According to this study, it is recommended that management exercise oversight and regulation over the concentrations of Ni, Cr, Cd, and As in the soil adjacent to asbestos mines, establish a designated control zone to restrict population activities, and locate residential zones at a safe distance from the asbestos mine production zone.