We studied Cd, Cr, As, Ni, Mn, Pb, and Hg in three agricultural areas of Hunan province and determined the potential non-carcinogenic and carcinogenic risks for residents. Soil and brown rice samples from Shimen, Fenghuang, and Xiangtan counties were analyzed by atomic absorption spectroscopy. Soil levels of Cd and Hg were greatest, followed by As and Ni. The mean concentrations of heavy metals in brown rice were Cd 0.325, Cr 0.109, As 0.344, Ni 0.610, Mn 9.03, Pb 0.023, and Hg 0.071 mg/kg, respectively. Cd and Hg had greater transfer ability from soil to rice than the other elements. Daily intake of heavy metals through brown rice consumption were estimated to be Cd 2.30, Cr 0.775, As 2.45, Ni 4.32, Pb 0.162, Mn 64.6 and Hg 0.503 µg/(kg·day), respectively. Cd, Hg and As Hazard Quotient values were greater than 1 and Cd, Cr, As and Ni Cancer Risk values were all greater than 10−4. The total non-carcinogenic risk factor was 14.6 and the total carcinogenic risk factor was 0.0423. Long-term exposure to heavy metals through brown rice consumption poses both potential non-carcinogenic and carcinogenic health risks to the local residents.