Neurodegenerative diseases, among which one of the more common is Alzheimer’s disease, are the one of the biggest global public health challenges facing our generation because of the increasing elderly population in most countries. With the growing burden of these diseases, it is essential to discover and develop new treatment options capable of preventing and treating them. Neurodegenerative diseases, among which one of the most common is Alzheimer’s disease, are a multifactorial disease and therefore demand multiple therapeutic approaches. One of the most important therapeutic strategies is controlling the level of acetylcholine—a neurotransmitter in cholinergic synapses—by blocking the degradation of acetylcholine using acetylcholinesterase inhibitors such as tacrine, galantamine, donepezil and rivastigmine. However, these drugs can cause some adverse side effects, such as hepatotoxicity and gastrointestinal disorder. Thus, the search for new, more effective drugs is very important. In the last few years, different active constituents from plants have been tested as potential drugs in neurodegenerative disease therapy. The availability, lower price and less toxic effects of herbal medicines compared with synthetic agents make them a simple and excellent choice in the treatment of neurodegenerative diseases. The empirical approach to discovering new drugs from the systematic screening of plant extracts or plant-derived compounds is still an important strategy when it comes to finding new biologically active substances. The aim of this review is to identify new, safe and effective compounds that are potential candidates for further in vivo and clinical tests from which more effective drugs for the treatment of Alzheimer’s disease could be selected. We reviewed the methods used to determine anti-Alzheimer’s disease activity. Here, we have discussed the relevance of plant-derived compounds with in vitro activity. Various plants and phytochemical compounds have shown different activity that could be beneficial in the treatment of Alzheimer’s disorders. Most often, medicinal plants and their active components have been investigated as acetylcholinesterase and/or butyrylcholinesterase activity inhibitors, modifiers of β-amyloid processing and antioxidant agents. This study also aims to highlight species with assessed efficacy, usable plant parts and the most active plant components in order to identify species and compounds of interest for further study. Future research directions are suggested and recommendations made to expand the use of medicinal plants, their formulations and plant-derived active compounds to prevent, mitigate and treat Alzheimer’s disease.