The accumulation of potentially toxic elements (PTEs) in agricultural soils is of particular concern in China, while its status, ecological risks, and human health hazards have been little studied in the permafrost areas of Northeast China. In this study, 75 agricultural soil samples (0–20 cm) were collected from the Arctic Village, Mo’he City, in the northernmost part of China. The average concentration (mean ± standard deviation) of As, Cd, Cr, Cu, Hg, Ni, Pb, and Zn were 12.11 ± 3.66 mg/kg, 0.11 ± 0.08 mg/kg, 52.50 ± 8.83 mg/kg, 12.08 ± 5.12 mg/kg, 0.05 ± 0.02 mg/kg, 14.90 ± 5.35 mg/kg, 22.38 ± 3.04 mg/kg, and 68.07 ± 22.71 mg/kg, respectively. Correlation analysis, cluster analysis, and principal component analysis indicated that As, Cu, Ni, and Zn likely originated from geogenic processes, Hg and Pb from long-range atmospheric transport, Cd from planting activities, and Cr from Holocene alluvium. The geo-accumulation index and enrichment factor showed that As, Cd, Hg, and Zn are enriched in soils. The Nemerow pollution index showed that 66.67%, 24%, and 1.33% of soil samples were in slight, moderate, and heavy pollution levels, respectively, with Hg being the most important element affecting the comprehensive pollution index. The potential ecological risk index showed that 48.00% and 1.33% of soil samples were in the moderate ecological risk and high potential ecological risk levels, respectively. The non-carcinogenic and carcinogenic human health risk index for adults and children were both less than 1, which was within the acceptable range. This study revealed the accumulation pattern of PTEs in agricultural soils of permafrost regions and provided a scientific basis for research on ecological security and human health.