This study assessed the significance and potential impact of potentially toxic element (PTE) (i.e., Mn, Pb, Cu, Zn, Cr, Cd, and Ni) pollution in the surface soil from an abandoned manganese mining area in Xiangtan City, Hunan Province, China, on the health of residents. The risks were sequentially evaluated using a series of protocols including: the geo-accumulation index (Igeo), pollution load index (PLI), potential ecological risk index (RI), and implications for human health from external exposures using the hazard quotient (HQ), hazard index (HI) and carcinogenic risk (CR). The results revealed that Mn and Cd were the major pollutants in the soil samples. The ecological risk assessment identified moderate risks, which were mainly derived from Cd (82.91%). The results of the health risk assessment revealed that generally across the area, the non-carcinogenic risk was insignificant, and the carcinogenic risk was at an acceptable level. However, due to local spatial fluctuation, some of the sites presented a non-carcinogenic risk to children. The soil ingestion pathway is the main route of exposure through both non-carcinogenic and carcinogenic risks, with Mn being the major contributor to non-carcinogenic risk, with Cr and Cd the major contributors to carcinogenic risk. In addition, three pollution sources were identified through the Pearson correlation coefficient and principal component analysis (PCA), which included: a. mining activities and emissions from related transportation; b. natural background; c. agricultural management practices and municipal sewage discharge. The study provides information on the effects of spatial variation for the development of the abandoned mining areas and a useful approach to the prioritization of locations for the development and utilization of soil in these areas in China.