Traditional Chinese medicine (TCM) has played a pivotal role in maintaining the health of people, and the intrinsic quality of TCM is directly related to the clinical efficacy. The medicinal ingredients of TCM are derived from the secondary metabolites of plant metabolism and are also the result of the coordination of various physiological activities in plants. Arbuscular mycorrhizal fungi (AMF) are among the most ubiquitous plant mutualists that enhance the growth and yield of plants by facilitating the uptake of nutrients and water. Symbiosis of AMF with higher plants promotes growth and helps in the accumulation of secondary metabolites. However, there is still no systematic analysis and summation of their roles in the application of TCM, biosynthesis and accumulation of active substances of herbs, as well as the mechanisms. AMF directly or indirectly affect the accumulation of secondary metabolites of TCM, which is the focus of this review. First, in this review, the effects of AMF symbiosis on the content of different secondary metabolites in TCM, such as phenolic acids, flavonoids, alkaloids and terpenoids, are summarized. Moreover, the mechanism of AMF regulating the synthesis of secondary metabolites was also considered, in combination with the establishment of mycorrhizal symbionts, response mechanisms of plant hormones, nutritional elements and expression of key enzyme their activities. Finally, combined with the current application prospects for AMF in TCM, future in-depth research is planned, thus providing a reference for improving the quality of TCM. In this manuscript, we review the research status of AMF in promoting the accumulation of secondary metabolites in TCM to provide new ideas and methods for improving the quality of TCM.