Alzheimer’s disease (AD) is a progressive neurodegenerative disorder characterized by memory loss and cognitive impairment. It severely affects the quality of life of victims. The prevalence of AD has been increasing in recent years. Therefore, it is of great importance to elucidate the pathogenic mechanism of AD and search for effective therapeutic approaches. Gut microbiota dysbiosis, an altered state of gut microbiota, has been well known for its involvement in the pathogenesis of AD. Much effort has been made in searching for approaches capable of modulating the composition of gut microbiota in recent years. Herbal medicines have attracted extensive attention in recent decades for the prevention and treatment of AD. Here, we gave an overview of the recent research progress on the modulatory effects of herbal medicines and herbal formulae on gut microbiota as well as the possible beneficial effects on AD, which may provide new insights into the discovery of anti-AD agents and their therapeutic potential for AD through modulating the composition of gut microbiota.