Investigation of the alkaloids from Peganum harmala seeds yielded two pairs of unique racemic pyrroloindole alkaloids, (±)-peganines A–B (1–2); two rare thiazole derivatives, peganumals A–B (3–4); six new β-carboline alkaloids, pegaharmines F–K (5–10); and 12 known analogues. Their structures, including stereochemistry, were elucidated through spectroscopic analyses, quantum chemistry calculations, and single-crystal X-ray diffraction. Notably, the incorporation of pyrrole and indole moieties in peganines A–B, thiazole fragments in peganumals A–B, and a C-1 α,β-unsaturated ester motif in pegaharmine F (5) are all rare, and their presence in the genus Peganum were demonstrated for the first time. All isolates were tested for antiproliferative activities against the HL-60, PC-3, and SGC-7901 cancer cell lines, and compounds 9, 11, 12, and 13 exhibited moderate cytotoxicity against HL-60 cancer cell lines with IC50 values in the range of 4.36–9.25 μM.

In this study, we screened 17 medicinal plants for binding activity to G-quadruplex d(TTGGGTT)4 by (1)H NMR spectroscopy and found that the crude extract of Peganum harmala L. seeds showed the most potential binding activity. Subsequently, (1)H NMR- and bioassay-guided isolation of the extract of P. harmala L. was performed to obtain four pairs of partially racemized β-carboline alkaloids, pegaharmines A-D (1-4). Their structures and absolute configurations were determined by extensive NMR analyses, X-ray crystallography, ECD calculations, and CD exciton chirality approaches. Interestingly, pegaharmine D (4), which showed the strongest G-quadruplex interaction, exhibited significant cytotoxic activity against three cancer cell lines. This work contributed a practical strategy for the discovery of novel G-quadruplex ligands from natural products and provided potential insights for using β-carboline alkaloids as anticancer lead compounds specifically targeting G-quadruplexes.