Autologous bone marrow mononuclear cell (BMMNC) transplantation has been widely studied in recent years. The fresh cell cocktail in BMMNCs, without going through the in vitro culture process, helps to establish a stable microenvironment for osteogenesis, and each cell type may play a unique role in bone regeneration. Our study compared the efficacy of concentrated fresh BMMNCs and cultured bone marrow-derived mesenchymal stem cells (BMSCs) in Beagle dogs for the first time. Fifteen-millimeter segmental bone defects were created in the animals' tibia bones. In BMMNCs group, the defects were repaired with concentrated fresh BMMNCs combined with β-TCP (n = 5); in cultured BMSC group, with in vitro cultured and osteo-induced BMSCs combined with β-TCP (n = 5); in scaffold-only group, with a β-TCP graft alone (n = 5); and in blank group, nothing was grafted (n = 3). The healing process was monitored by X-rays and single photon emission computed tomography. The animals were sacrificed 12 months after surgery and their tibias were harvested and analyzed by microcomputed tomography and hard tissue histology. Moreover, the microstructure, chemical components, and microbiomechanical properties of the regenerated bone tissue were explored by multiphoton microscopy, Raman spectroscopy and nanoindentation. The results showed that BMMNCs group promoted much more bone regeneration than cultured BMSC group. The grafts in BMMNCs group were better mineralized, and they had collagen arrangement and microbiomechanical properties similar to the contralateral native tibia bone. These results indicate that concentrated fresh bone marrow mononuclear cells may be superior to in vitro expanded stem cells in segmental bone defect repair.