Gold mining is associated with serious heavy metal pollution problems. However, the studies on such pollution caused by gold mining in specific geological environments and extraction processes remain insufficient. This study investigated the accumulation, fractions, sources and influencing factors of arsenic and heavy metals in the sediments from a gold mine area in Southwest China and also assessed their pollution and ecological risks. During gold mining, As, Sb, Zn, and Cd in the sediments were affected, and their accumulation and chemical activity were relatively high. Gold mining is the main source of As, Sb, Zn and Cd accumulation in sediments (over 40.6%). Some influential factors cannot be ignored, i.e., water transport, local lithology, proportion of mild acido-soluble fraction (F1) and pH value. In addition, arsenic and most tested heavy metals have different pollution and ecological risks, especially As and Sb. Compared with the other gold mining areas, the arsenic and the heavy metal sediments in the area of this study have higher pollution and ecological risks. The results of this study show that the local government must monitor potential environmental hazards from As and Sb pollution to prevent their adverse effects on human beings. This study also provides suggestions on water protection in the same type of gold-mining areas.