Herbal medicines are widely used for clinical purposes worldwide. These herbs are susceptible to phytopathogenic fungal invasion during the culturing, harvesting, storage, and processing stages. The threat of fungal and mycotoxin contamination requires the evaluation of the health risks associated with these herbal medicines. In this study, we collected 138 samples of 23 commonly used herbs from 20 regions in China, from which we isolated a total of 200 phytopathogenic fungi. Through morphological observation and ITS sequencing, 173 fungal isolates were identified and classified into 24 genera, of which the predominant genera were Fusarium (27.74%) and Alternaria (20.81%), followed by Epicoccum (11.56%), Nigrospora (7.51%), and Trichocladium (6.84%). Quantitative analysis of the abundance of both Fusarium and Alternaria in herbal medicines via RT-qPCR revealed that the most abundant fungi were found on the herb Taraxacum mongolicum, reaching 300,000 copies/μL for Fusarium and 700 copies/μL for Alternaria. The in vitro mycotoxin productivities of the isolated Fusarium and Alternaria strains were evaluated by using liquid chromatography–tandem mass spectrometry (LC-MS/MS), and it was found that the Fusarium species mainly produced the acetyl forms of deoxynivalenol, while Alternaria species mainly produced altertoxins. These findings revealed widely distributed fungal contamination in herbal medicines and thus raise concerns for the sake of the quality and safety of herbal medicines.