In order to achieve the demands of multi-beam splitting in the optical communication system, a compact high-efficiency silica-based nine-channel beam splitter grating (BSG) at infrared wavelength is proposed based on a three-layer array structure. The grating profile parameters are investigated by the rigorous coupled-wave analysis (RCWA) and the finite-element method (FEM). The total efficiency of the nine-channel beam splitter reaches 96.98%, and the error of uniformity is reduced to 2.98%. The results show that the theoretical values of the two optimization methods are in good agreement. Moreover, the presented grating has a good performance against fabrication tolerances. The modal method is used for detailed physical explanation of the energy exchange and propagation modes inside the grating. The proposed high-efficiency nine-channel beam separation by three-layer arrays should be helpful for developing the multi-beam splitting of the optical system of the lithography machine and the new ultra-precision optical devices.