A field survey was conducted to determine the pollution grade, sources, potential ecological risk, and health risk of soil potentially toxic elements (PTEs) in Xikuangshan Mine (XKS), the largest antimony (Sb) deposit in the world. A total of 106 topsoil samples were collected from 6 sites in XKS to measure the concentrations of PTEs Cr, Zn, Cd, Pb, As, Hg, and Sb. The results show that the average concentrations of these elements at all six sites were generally greater than their corresponding background values in Hunan province, especially Sb, Hg, and As. Correlation and principal component analyses suggested that Cd, Zn, Pb, Hg, and Sb were primarily released from mining and other industrial and human activities, while Cr and As were mainly impacted by the parent material from pedogenesis. A risk index analysis showed that, overall, sites were at very high ecological risk, and Sb is the highest ecological risk factor, followed by Cd and Hg. According to health risk assessment, oral ingestion is the main non-carcinogenic and carcinogenic risk exposure route. The higher potentially non-carcinogenic and carcinogenic risks happen to the local children who live in the vicinity of mining area. It revealed that the mining and smelting processes of XKS have negatively influenced the local people, therefore, we should pay increasing attention to this practical issue and take effective measures to protect the ecology of XKS.