Alzheimer's disease (AD) is a severe condition in aging societies. Although research on this disease is advancing rapidly, thus far few very effective drugs are available for AD patients. The currently widely used medicines such as donepezil and galantamine transiently improve the symptoms of patients with mild to moderate AD. They are hardly capable of preventing, halting or reversing the progression of this disease. In the long history of development of traditional Chinese medicine, many herbs have been discovered and employed to treat dementia diseases in clinics in China. In recent decades, a number of agents were isolated from these herbs and their efficacies against AD were tested. Some flavonoids, alkaloids, phenylpropanoids, triterpenoid saponins, and polysaccharides were demonstrated to have potential efficacies against AD via targeting multiple pathological changes of this disease. In this article, we reviewed research progress on the efficacies and underlying mechanisms of these agents.