The aim of this study was to investigate the levels of trace elements in olive oils from different locations and their use for geographical authentication. Concentrations of seventeen elements were determined in a total of 42 olive oils from Tunisia, Spain (Basque country), and southern France, and in nine soil samples from Tunisia by quadrupole inductively plasma mass spectrometry. The compilation of appropriate techniques integrated into the analytical procedure achieved a precision (RSD) between 2% and 15% and low limits of detection (between 0.0002 and 0.313 µg kg−1). The accuracy of the analytical method applied for olive oil analysis was evaluated using SRM NIST 2387 Peanut butter. The recoveries obtained after microwave-assisted digestion for the certified elements ranged between 86% and 102%. Concentrations of non-certified elements (V, Cr, Co, Ni, Ba, Rb, Sr, Cd, Pb, and As) were presented. The use of Pearson correlation applied on paired Tunisian oil/soil samples has shown that several elements (Mg, Mn, Ni, and Sr) were significantly correlated. The multivariate statistics using principal component analysis have successfully discriminated against three studied origins. The most significant variables were the elemental concentrations of Cu, Cr, Fe, Mn, Sr, V, and Zn. This study shows the potential of applying trace elements profiles for olive oil geographical discrimination.