Potentially toxic elements (PTEs) in Chinese agricultural soils, including those in some heritage protection zones, are serious and threaten food safety. Many scientists think that these PTEs may come from parent rock. Hence, at a karst rice-growing agricultural heritage area, Babao town, Guangnan County, Yunnan Province, China, the concentrations of eight PTEs (As, Cd, Cr, Cu, Hg, Ni, Pb, and Zn) were determined in 148 surface soil, 25 rock, and 52 rice grain samples. A principal component analysis (PCA) and hierarchical cluster analysis were used to divide the surface soil into groups, and inverse distance weighting (IDW) was used to analyze the spatial distribution of PTEs. Soil pollution was assessed with the geoaccumulation index (Igeo). The results show that Cd, Hg, Zn, and Cr were polluting the soil (average Igeo > 0). The highest concentration of PTEs was distributed in the southwest of Babao town in the carbon rock area, which had the highest pH and soil total organic carbon (Corg), Mn, and TFe2O3 contents. PCA biplots of soil samples showed that the carbon rock area was associated with the most species of PTEs in the study area including Pb, Cd, Hg, As, and Zn. The clastic rock area was associated with Cu and Ni, and the lime and cement plants were associated with CaO, pH, Corg, TC, and aggravated PTE pollution around factories. In high-level PTE areas, rice was planted. Two out of 52 rice grain samples contained Cd and 4 out of 52 rice grain samples had Cr concentrations above the Chinese food safety standard pollutant limit (Cd 0.2 mg/kg; Cr 1 mg/kg). Therefore, the PTEs from parent rocks are already threatening rice safety. The government should therefore plan rice cultivation areas accordingly.