The dysfunction of bone marrow mesenchymal stem cells (BMSCs) in osteogenic differentiation is one of the main causes of age‐related bone loss. Our previous studies have shown that low‐magnitude vibration (LMV) induces the osteogenic differentiation of BMSCs derived from ovariectomized osteoporotic rats. To investigate whether LMV promotes osteogenic differentiation of BMSCs and its underlying mechanisms in aged rats, 20‐month‐old female Sprague‐Dawley rats (n = 20) were randomly divided into LMV group (rats were vibrated at 0.3 g and 90 Hz for 30 minutes, once daily, 5 days a week until 12 weeks for subsequent analysis, n = 10), static group (rats were placed in the box on the vibration platform without vibration, n = 10); 6‐month‐old female Sprague‐Dawley rats were used as control (young group, n = 10). The bone mineral density and bone strength of aged rats were significantly decreased compared with the young rats. Furthermore, the primary BMSCs isolated and cultured from the aged rats with the whole‐bone marrow differential pasting method showed a decreased ability in osteogenic differentiation compared with that from the young rats. Then the differentially expressed miRNAs between the aged and young rat‐derived BMSCs were screened by high‐throughput sequencing and verified by qRT‐PCR, and we found that miR‐378a‐3p was significantly downregulated in the aged rat‐derived BMSCs compared with the young rat‐derived BMSCs. By transfecting miRNA mimics and inhibitors, miR‐378a‐3p was confirmed to promote the expression levels of osteogenic genes (Runx2, ALP, Col I, and OCN) and ALP activity of the aged rat‐derived BMSCs. Meanwhile, the expression levels of osteogenic genes and miR‐378a‐3p of aged rat‐derived BMSCs were significantly upregulated by LMV (cells were vibrated at 0.3 g and 90 Hz for 30 minutes a day, until 5 days for subsequent analysis), while the LMV‐induced osteogenic gene expression levels of aged rat‐derived BMSCs were suppressed by miR‐378a‐3p inhibitors. Furthermore, the inhibition of growth factor receptor‐bound protein 2 (Grb2) by miR‐378a‐3p and Grb2‐siRNA promoted the LMV‐induced osteogenic differentiation of aged rat‐derived BMSCs. Additionally, LMV was found to promote bone mineral density and bone strength of aged rats in vivo, as well as upregulating the expression level of miR‐378a‐3p and downregulating the expression level of Grb2 of BMSCs from aged rats. These results suggest that LMV induces osteogenic differentiation of BMSCs through miR‐378a‐3p/Grb2 pathway to improve bone mineral density and mechanical properties in a rat model of age‐related bone loss.