A novel Al2O3/SiO2 core–shell composite abrasive was successfully synthesized via a facile sol–gel strategy and utilized to promote the surface quality and polishing efficiency of sapphire wafer. The silica shell with dense and amorphous structure was successfully coated on Al2O3 core. Its synthesis mechanism was discussed deeply, and good dispersibility and uniform particle size were obtained. The acquired core–shell composite particles were employed for the planarization machining of sapphire wafer. Results show that the decrease in surface roughness of Al2O3/SiO2 core–shell composite abrasives during polishing is more than 20.2% of that of pure Al2O3 abrasives, the material removal rate is more than 5.1% of that of pure Al2O3 abrasives, and the residual stress of the sapphire wafer finished by composite abrasives is decreased by approximately 40%. These results demonstrate that the Al2O3/SiO2 core–shell composite abrasives provide a more excellent polishing performance for the sapphire wafer, resulting in smoother and scratch-free surface with a higher material removal rate compared with pure Al2O3 abrasives. On the basis of the reaction product analysis and elastic-plastic micro-contact mechanics, the reasons for the improvement of polishing performance were revealed in terms of chemical corrosion and mechanical abrasion.