Osteoporosis (OP) is a common and complex chronic metabolic disease with an increasing incidence rate, which has markedly increased the human health burden worldwide. The predominant cause of OP is an imbalance between osteoblasts (OB) and osteoclasts (OC). Studies on the correlation between bone marrow-derived mesenchymal stem cells (BMSCs) and OP have indicated that BMSCs-induced OB differentiation is an important pathway for bone tissue renewal. Chinese medicinal herbs have been used for centuries to treat various types of OPs because they are safer and more effective. The in vivo and in vitro experiments have confirmed that these herbs or their primary phytochemicals may exert therapeutic effects by stimulating BMSCs differentiation, which restores OB and OP balance, inhibits adipocyte differentiation, exerts anti-inflammatory and antioxidant effects, regulates the immune system, etc. This review summarizes the research on how Chinese medicinal herbs or their primary phytochemicals treat OP by stimulating BMSC differentiation and provides a scientifically reliable basis and perspective for their future clinical application.