Nanostructured hydroxyapatite (HA) is a new class of biocompatible fillers which has been recently utilized in bio hybrid materials by virtue of its excellent tissue bioactivity and biocompatibility. However, the need for higher thermal stability, solubility, surface bioactivity, radiopacity, and remineralization ability suggests a divalent cation substitution of HA for use in light curable dental restorative composites. In this work, structural and optical properties of Sr-doped hydroxyapatite were studied using first-principle calculations based on density functional theory (DFT). Next, Sr-doped hydroxyapatite (HA) was prepared via a new ionic liquid-assisted hydrothermal (ILH) route. Samples were characterized using X-ray diffraction (XRD), scanning electron microscopy (SEM)/energy dispersive spectroscopy (EDS), Fourier transform infrared spectroscopy (FTIR), transmission electron microscopy (TEM), dynamic light scattering (DLS), Brunauer–Emmett–Teller (BET) surface area analysis, and cell viability. The obtained experimental data showed that the nucleation and crystal growth process controlled by [BMIM]Br molecules results in uniform products with small and regular particles and high specific surface areas. Finally, cytotoxicity tests showed that the as-prepared Sr-doped HA nanoparticles have good biocompatibility (≥91%), confirming their potential for use in photo-curable dental restorative composites.