Peganum harmala L. is a perennial herb of the Tribulus family and its aerial parts and seeds can be used as medicine in the traditional medicine of China. However, the differences in chemical components and antibacterial activity between different parts have not been reported. In this study, the chemical composition of the different parts of P. harmala was characterized by high-performance liquid chromatography (HPLC) and headspace-solid phase microextraction-gas chromatography-mass spectrometry (HS-SPME-GC-MS). The antimicrobial activities of the different parts and some isolated components were also carried out on 12 bacterial strains and phytopathogenic fungi. The HPLC results revealed that the contents of harmine and harmaline in the seeds were higher than that in the aerial parts. A total of 94 volatile organic compounds (VOCs) were tentatively identified by HS-SPME-GC-MS for the first time. The major components were methyl hexadecanoate, p-xylene, octane, (Z)-9-octadecanoate, ethylbenzene, methyl octadecanoate, ethyl hexadecanoate, and methyl tetradecanoate. At the concentration of 800 μg·mL−1, the methanol extracts of seeds showed stronger antimicrobial activities with a wide antimicrobial spectrum, inhibiting Escherichia coli (ATCC 24433), Xanthomonas oryzae (ACCC 11602), and Xanthomonas axonopodis with inhibitory rates of more than 90%. Furthermore, harmine and harmaline showed better antibacterial activities against all the bacteria. These findings indicated that alkaloids from P. harmala could account for antimicrobial activity, which could be used as lead molecules in the development of new antimicrobial drugs.