Due to the irreversibility of heavy metal pollution, the presence of heavy metals in farmland soil is associated with severe ecological risks that endanger both the environment and human health. Cadmium (Cd) and mercury (Hg) are two toxic heavy metals found widely in polluted soil. Cd is not readily fixed in the soil and is therefore easily accumulated by plants, while Hg has a wide range of pollution sources. The aims of this study were to explore the spatial variation in Cd and Hg concentrations in farmland soil in Poyang Lake Plain, China, and to assess their potential ecological risks as influenced by natural and human factors. A total of 283 soil samples were obtained from Fengcheng city, central Jiangxi Province. Data were then analyzed using geostatistics, the potential ecological risk index, Pearson’s correlation analysis, and Geodetector. The results showed moderate variation in soil Cd and Hg concentrations, with a remarkable difference in their spatial distribution. Cd concentrations in the northwest and northeast of Fengcheng were below the regional background level in Jiangxi; in most remaining areas, Cd concentrations were between the regional background level and national risk screening value. Areas with Hg concentrations lower than the regional background level were largely concentrated in the south, east and north of Fengcheng, and gradually increased towards the central, where they exceeded the regional background level but were below the national risk screening value. Overall, the potential ecological risk level of Cd was predominantly low, while that of Hg was moderate. The comprehensive potential ecological risk was low in most areas for both Cd and Hg, with some scattered areas of moderate risk. Moreover, the comprehensive potential ecological risk index of both Cd and Hg was significantly correlated with soil pH, total phosphorous, elevation, distance from a river (p < 0.01), and distance from a road (p < 0.05). The most significant factor influencing the comprehensive potential ecological risk index of these two heavy metals was soil pH of 5.2–5.6, followed by total p ≤ 0.52 mg kg−1. In conclusion, moderate pollution of Cd and Hg occurred in farmland soil in Poyang Lake Plain where their comprehensive potential ecological risk level was generally low and mainly influenced by soil pH and total phosphorous.