Pyrrolizidine alkaloids (PAs) occur as contaminants in plant-based foods and herbal medicines. Following metabolic activation by cytochrome P450 (CYP) enzymes, PAs induce DNA damage, hepatotoxicity and can cause liver cancer in rodents. There is ample evidence that the chemical structure of PAs determines their toxicity. However, more quantitative genotoxicity data are required, particularly in primary human hepatocytes (PHH). Here, the genotoxicity of eleven structurally different PAs was investigated in human HepG2 liver cells with CYP3A4 overexpression and PHH using an in vitro test battery. Furthermore, the data were subject to benchmark dose (BMD) modeling to derive the genotoxic potency of individual PAs. The cytotoxicity was initially determined in HepG2-CYP3A4 cells, revealing a clear structure–toxicity relationship for the PAs. Importantly, experiments in PHH confirmed the structure-dependent toxicity and cytotoxic potency ranking of the tested PAs. The genotoxicity markers γH2AX and p53 as well as the alkaline Comet assay consistently demonstrated a structure-dependent genotoxicity of PAs in HepG2-CYP3A4 cells, correlating well with their cytotoxic potency. BMD modeling yielded BMD values in the range of 0.1–10 µM for most cyclic and open diesters, followed by the monoesters. While retrorsine showed the highest genotoxic potency, monocrotaline and lycopsamine displayed the lowest genotoxicity. Finally, experiments in PHH corroborated the genotoxic potency ranking, and revealed genotoxic effects even in the absence of detectable cytotoxicity. In conclusion, our findings strongly support the concept of grouping PAs into potency classes and help to pave the way for a broader acceptance of relative potency factors in risk assessment.