The concentrations, chemical availability, distribution, and sources of potentially toxic elements (PTEs) in the soil of Xiangjiang Basin in Hunan Province, China were investigated at 85 sites. The highest mean concentrations of Cd, Cu, Zn, As, and Pb were observed in Hengyang, whereas those for Mn, Co, and Hg were observed in Changde. The pollution index values followed the order: Cd > Hg > Cu > Zn > As > Pb; the mean geo-accumulation index values were in the order: Cd > Hg > Pb > Cu > Zn > As > Co > Mn. Cd was associated with moderate contaminated level, Hg and Pb were associated with moderate contaminated to uncontaminated level, and Cu, Zn, As, Co, and Mn were associated with uncontaminated level of pollution. Furthermore, 64.5% of Cd was water-soluble and exhibited exchangeable fractions; its chemical availability posed a risk to the ecosystem. Spatial analysis, principal component analysis, and a positive matrix factorization model were used to assess the PTE sources. Four principal components contributed to 88.8% of the 8 PTEs concentrations. Mining, smelting, industrial, and agricultural activities, alongside sewage irrigation, the use of agrochemicals, and vehicular emissions are the possible anthropogenic sources that pollute agricultural products and threaten human health in the Xiangjiang Basin.