Sexual enhancers increase sexual potency, sexual pleasure, or libido. Substances increasing libido alter the concentrations of specific neurotransmitters or sex hormones in the central nervous system. Interestingly, the same pathways are involved in the mechanisms underlying many psychiatric and neurological disorders, and adverse reactions associated with the use of aphrodisiacs are strongly expected. However, sexual enhancers of plant origin have gained popularity over recent years, as natural substances are often regarded as a safer alternative to modern medications and are easily acquired without prescription. We reviewed the psychiatric and neurological adverse effects associated with the consumption of herbal aphrodisiacs Areca catechu L., Argemone Mexicana L., Citrus aurantium L., Eurycoma longifolia Jack., Lepidium meyenii Walp., Mitragyna speciosa Korth., Panax ginseng C. A. Mey, Panax quinquefolius L., Pausinystalia johimbe (K. Schum.) Pierre ex Beille, Piper methysticum G. Forst., Ptychopetalum olacoides Benth., Sceletium tortuosum (L.) N. E. Brown, Turnera diffusa Willd. ex. Schult., Voacanga africana Stapf ex Scott-Elliot, and Withania somnifera (L.) Dunal. A literature search was conducted on the PubMed, Scopus, and Web of Science databases with the aim of identifying all the relevant articles published on the issue up to June 2020. Most of the selected sexual enhancers appeared to be safe at therapeutic doses, although mild to severe adverse effects may occur in cases of overdosing or self-medication with unstandardized products. Drug interactions are more concerning, considering that herbal aphrodisiacs are likely used together with other plant extracts and/or pharmaceuticals. However, few data are available on the side effects of several plants included in this review, and more clinical studies with controlled administrations should be conducted to address this issue.