Osteoporosis (OP) is a systemic skeletal disease prevalent in older adults, characterized by substantial bone loss and deterioration of microstructure, resulting in heightened bone fragility and risk of fracture. Traditional Chinese Medicine (TCM) herbs have been widely employed in OP treatment owing to their advantages, such as good tolerance, low toxicity, high efficiency, and minimal adverse reactions. Increasing evidence also reveals that many plant-based compounds (or secondary metabolites) from these TCM formulas, such as resveratrol, naringin, and ginsenoside, have demonstrated beneficial effects in reducing the risk of OP. Nonetheless, the comprehensive roles of these natural products in OP have not been thoroughly clarified, impeding the development of synergistic formulas for optimal OP treatment. In this review, we sum up the pathological mechanisms of OP based on evidence from basic and clinical research; emphasis is placed on the in vitro and preclinical in vivo evidence-based anti-OP mechanisms of TCM formulas and their chemically active plant constituents, especially their effects on imbalanced bone homeostasis regulated by osteoblasts (responsible for bone formation), osteoclasts (responsible for bone resorption), bone marrow mesenchymal stem cells as well as bone microstructure, angiogenesis, and immune system. Furthermore, we prospectively discuss the combinatory ingredients from natural products from these TCM formulas. Our goal is to improve comprehension of the pharmacological mechanisms of TCM formulas and their chemically active constituents, which could inform the development of new strategies for managing OP.