Azurite, a historical blue mineral pigment, has previously been described to contain certain elemental impurities. These may originate from host rocks, vein fillings, or the primary copper ore mineralization. In this study, azurites (and also green malachites) from three important Central European deposits with a potential of being exploited for pigment usage already in the Middle Ages have been studied, together with azurite from Chessy, France, with a different geological setting. Using electron probe microanalysis and, more importantly, laser ablation inductively coupled plasma mass spectroscopy for trace elemental analysis, several indicators were pinpointed as important for provenance: characteristic elemental fingerprint of the deposit, e.g., elevated lead (Pb) in combination with rare earth elements, may be combined with zinc (Zn)/arsenic (As) ratio (indicating sources of excess Zn in the primary deposit) and the overall amount of metal impurities (suggesting the source mineral of copper for azurite formation). In addition, malachites from the same deposits were found to preferentially incorporate primary ore metal elements as well as Cd, Mg, Mn, or U. Therefore, if azurite pigment contains an elevated amount of malachite as an impurity, it may significantly influence the overall elemental composition. The results obtained on geological samples were applied to two micro-samples of works of art containing azurite-rich layers originating from the 13th–14th and 16th centuries. It was shown that it is highly beneficial to focus on the overall trace elemental composition of the paint layer and not on the admixed mineral grains, as their presence, especially in minute micro-samples, is largely accidental and thus not representative. Although a higher number of samples need to be studied in the future, the newly described criteria made it possible to exclude some of the localities of the employed azurite pigment. This confirmed the key importance of trace elements analysis of mineral pigments for the provenance studies of fine arts.