Arable land soil is one of the most precious natural resources of Earth, it provides the fundamental material and numerous resources essential for the development of human society. To determine the pollution of potential toxic factors in the surface soil of cultivated land and its risks to human health, concentrations of five different potentially toxic elements (PTEs) were detected in 1109 soil samples collected in Xiangzhou, China, in 2019. In this study, health risk assessment was used to judge the degree of pollution in the study area, the result of Geographic Information System (GIS) was as used to research the spatial distribution characteristics of PTEs, and random forest (RF) was used to assess the natural and man-made influencing factors. We investigated the sources of PTEs through quantifying the indicators, which gave further insights. The main results are: (1) In arable land soil, the average content of PTEs is 0.14 mg/kg cadmium (Cd), 0.05 mg/kg mercury (Hg), 12.89 mg/kg arsenic (As), 29.23 mg/kg lead (Pb), and 78.58 mg/kg chromium (Cr), respectively. The content of As and Pb outpaced the background value of Hubei soil. (2) The human health risk assessment in Xiangzhou indicates that the most important exposure pathway is soil ingestion, occupied about 99% to health risks of PTEs; non-carcinogenic risk from exposure to As, Pb and Cr in soil was higher than the limit (overall potential risk index, HI > 1) for both children and adults. Moreover, carcinogenic risk postured by Cd, Cr, and As was higher than the limit (10−4) through soil exposure for both children and adults, indicating that Cd, As, Pb and Cr in soil have significant effect on people’s health through exposure. (3) We found that the increased PTEs in the arable land soil mainly originated from potential water sources, air and soil pollution sources, breeding farms, and mining areas.