Geochemical indices used to identify the emerging anomalies of potentially harmful elements in topsoil depend on background values (BVs). For urban sites, it is reasonable to estimate native BVs through the targeted selection of peri-urban sampling sites or by distinguishing a useful background subset (BS) within the peri-urban dataset. Here, the goals were to examine the influence of Quaternary deposits on various types of topsoil variables, identify the variables most helpful for cluster analysis intended for the choice of background subset (BS), and compare background values (BVs) based on different background subsets. Composite topsoil samples from a peri-urban area were used for the determination of the following variables: contents of 26 elements and components of the bulk mineralogical composition, as well as the sand, silt, and clay fractions and loss-on-ignition (LOI) at 550 °C and at 950 °C. Although Quaternary lithology influences topsoil elemental contents or granulometric fractions, percentages of illite, kaolinite, orthoclase, quartz, albite, dolomite, and LOI at 550 °C, the choice of BS, according to it, is not recommended, as BVs based on topsoil texture are superior. However, cluster analysis using topsoil fractions < 2, <63, and >63 μm or the contents of Al, Fe, K, Ti, Ga, Nb, Rb, and Si are preferable. It is recommended to use these reference variables for the selection of BS.