Mining activities are one of the main contamination sources of Cd in soil. However, the information about the influence of silver mining on Cd pollution in soil in mining-affected areas is limited. In the present study, sixteen paired soil and rice grain samples were collected from the farmland along the Luxi River nearby a silver mine in Yingtan City, Jiangxi Province, China. The total, bioavailable, and fraction of Cd in soil and Cd content in rice grain were determined by inductively coupled plasma mass spectrometry. The transformation of Cd in the soil–rice system and potential health risk via consumption of these rice grains were also estimated. The results showed that Cd concentration in these paddy soils ranged from 0.21 to 0.48 mg/kg, with the mean Cd concentration (0.36 mg/kg) exceeded the national limitation of China (0.3 mg/kg, GB 15618-2018). Fortunately, all these contaminated paddy soils were just slightly polluted, with the highest single-factor pollution index value of 1.59. The DTPA- and CaCl2-extractable Cd in these paddy soils ranged from 0.16 to 0.22 mg/kg and 0.06 to 0.11 mg/kg, respectively, and the acid-soluble Cd occupied 40.40% to 52.04% of the total Cd, which was the highest among different fractions. The concentration of Cd in rice grain ranged from 0.03 to 0.39 mg/kg, and the mean Cd concentration in rice grain (0.16 mg/kg) was within the national limitation of China (0.2 mg/kg, GB 2762-2017). The bioaccumulation factor of Cd in rice grain ranged from 0.09 to 1.18, and its correlation with various indicators was nonsignificant (p < 0.05). Health risk assessment indicated that the noncarcinogenic risk for local rice consumers was within the acceptable range, but the carcinogenic risk (CR) was ranging from 1.24 × 10−2 to 1.09 × 10−3 and higher than the acceptable range (1.0 × 10−4), indicating that the local rice consumers suffered serious risk for carcinogenic diseases. The results of the present study can provide reference for safety production of rice in silver mining-affected areas.