Heavy metals as environmental pollutants can have natural or anthropogenic origin. To determine the river sediment pollution status, it is crucial to have appropriate reference samples, free of anthropogenic impact, and natural reference samples should be used wherever and whenever possible. The collection of reference samples should be performed in the vicinity of the research area in a place that belongs to the same geological environment and is undisturbed by human activity. The main purpose of this study was to compare concentrations of heavy metals from different rivers with background values to show that the usage of natural background values is the best option when assessing pollution status, but also to underline that the natural background values have to correspond to the analyzed sediments. In this study, 5 river sediments from Sava, 17 from Great War Island (GWI), 11 from Danube, 24 from Tisa, 47 from Tamiš, and 11 from Timok were evaluated relative to reference samples from the Sava and Tisa Rivers. The results indicate that geological origin has a strong influence on the content of heavy metals in river sediments, primarily regarding concentrations of Ni and Co. Furthermore, Tamiš, Tisa, Sava, and Danube sediments are under strong anthropogenic influence.