In recent years, near-infrared (NIR) phosphors have attracted great research interest due to their unique physical properties and broad application prospects. However, obtaining NIR phosphors with both high quantum efficiency and excellent thermal stability remains a great challenge. In this study, novel NIR Ca3Mg2ZrGe3O12:Cr3+ phosphors were successfully prepared using a high-temperature solid-phase method, and the structure and luminescent properties of the material were systematically investigated. Ca3Mg2ZrGe3O12:0.01Cr3+ emits NIR light in the range of 600 to 900 nm with a peak at 758 nm and a half-height width of 89 nm under the excitation of 457 nm blue light. NIR luminescence shows considerable quantum efficiency, and the internal quantum efficiency of the optimized sample is up to 68.7%. Remarkably, the Ca3Mg2ZrGe3O12:0.01Cr3+ phosphor exhibits a near-zero thermal quenching behavior, and the luminescence intensity of the sample at 250 °C maintains 92% of its intensity at room temperature. The mechanism of high thermal stability has been elucidated by calculating the Huang Kun factor and activation energy. Finally, NIR pc-LED devices prepared from Ca3Mg2ZrGe3O12:0.01Cr3+ phosphor with commercial blue LED chips have good performance, proving that this Ca3Mg2ZrGe3O12:0.01Cr3+ NIR phosphor has potential applications in night vision and biomedical imaging.