Radiological aspects such as natural radioactivity of 238U, 232Th, 226Ra, 40K combined with potentially toxic metal(loid) (PTM) distribution features were seldom simultaneously investigated in rare earth element (REE) processing activities. This work was designed to investigate the distribution levels of natural radioactivity, air-absorbed dose rate of γ radiation as well as PTMs at a typical REE plant in Guangdong, China. Ambient soils around REE processing facilities were sampled, measured and assessed. The natural radioactivity of radionuclides of the samples was determined using a high-purity germanium γ-energy spectrometer while the air-absorbed dose rate of γ radiation was measured at a height of 1 m above the ground using a portable radiometric detector. The PTM content was measured by inductively coupled plasma mass spectrometry (ICP-MS). The results showed that the specific activities of the radionuclides ranged from 80.8 to 1990.2, 68.2 to 6935.0, 78.4 to 14,372.4, and 625.4 to 2698.4 Bq·kg−1 for 238U, 226Ra, 232Th, and 40K, respectively, representing overwhelmingly higher activity concentrations than worldwide soil average natural radioactivity. The radium equivalent activity and external hazard index of most samples exceeded the limits of 370 Bq·kg−1 and 1, respectively. The measured air-absorbed dose rate of γ radiation was in a range of 113~4004 nGy·h−1, with most sites displaying comparatively higher values than that from some other REE-associated industrial sites referenced. The content levels of PTMs of Cu, Ni, Zn, Mn, Pb, Cd, Cr, and As were 0.7~37.2, 1.8~16.9, 20.4~2070.5, 39.4~431.3, 2.3~1411.5, 0.1~0.7, 6.7~526.1, and 59.5~263.8 mg·kg−1, respectively. It is important to note that the PTM contents in the studied soil samples were 2.1~5.4 times higher for Zn-As and 1.4 times higher for Pb than the third level of the China soil standard while 2.5~13 times higher for Zn-As and 1.2 times higher for Pb than Canadian industry standard. The findings call for subsequent site remediation to secure the ecological environment and human health after the REE processing plant was decommissioned.