Although the novel root-end filling material containing zirconium oxide (NRFM-Zr) which is hydroxyapatite-based may promote osteoblast differentiation, the molecular mechanism remains unclear. The aim of this study is to investigate it underlying the osteogenic/odontogenic differentiation of human osteosarcoma MG-63 cells induced by NRFM-Zr, compared with calcium silicate-based mineral trioxide aggregate (MTA), and glass ionomer cement (GIC). Firstly, three different types of root filling materials were co-cultured with MG-63 cells, and their cell toxicity, alkaline phosphatase (ALP) activity, and calcium ion concentration were evaluated. Next, gene expression profiling microarray was employed to analyze the impact of the materials on the gene expression profile of MG-63 cells. The results of cell viability revealed that NRFM-Zr group had no significant difference compared to the negative control group. After 5 and 7 days of cultivation, both the NRFM-Zr and MTA groups exhibited significantly higher ALP activity compared to the negative control (p < 0.05). Moreover, the NRFM-Zr group had the highest calcium ion concentration, while the GIC group was the lowest (p < 0.05). Gene expression profiling microarray analysis identified 2915 (NRFM-Zr), 2254 (MTA) and 392 (GIC) differentially expressed genes, respectively. GO functional and KEGG pathway analysis revealed that differentially expressed genes of NRFM-Zr, MTA and GIC participated in 8, 6 and 0 differentiation-related pathways, respectively. Comparing the molecular mechanisms of osteogenic/odontogenic differentiation induced by hydroxyapatite-based NRFM-Zr and calcium silicate-based MTA, it was found that they shared similarities in their molecular mechanisms of promoting osteogenic differentiation. NRFM-Zr primarily promotes differentiation and inhibits cell apoptosis, thereby enhancing osteogenic/odontogenic differentiation of MG-63 cells. Furthermore, the inducing efficacy of NRFM-Zr was found to be superior to MTA.