The Operational Transfer Path Analysis (OTPA) is widely used to analyze signal transfer paths in various fields, but its applications in gear transmission systems are limited. In this paper, the OTPA method is appropriately applied to the study of vibration signal transfer paths of fixed shaft gear transmission systems. Firstly, the dynamic model of a three-axis two-stage fixed-shaft gear transmission system is developed. This model takes into account factors such as the impact of dynamic center distance on backlash and provides a comprehensive analysis of time-varying meshing stiffness and forces. Experiments are conducted to verify the accuracy of the model. Subsequently, this paper delves into the OTPA process for solving the signal transfer coefficient matrix. It also derives the calculation method for computing the contribution of signal transfer paths, which provides a theoretical basis for establishing the evaluation mechanism of the signal transfer paths. Next, the gear transmission system’s signal transfer is analyzed based on the OTPA method, and the contribution of each path is computed. The dominant path is identified by ranking the contribution of these paths. Finally, the excitation experiment of a single signal transfer path reveals the essence of the signal transfer mechanism. This research contributes to a deeper understanding of signal transfer paths within gear transmission systems based on the OTPA method, it identifies the optimal distribution of sensor installation positions for fault signal diagnosis, paving the way for more effective fault detection and diagnosis in the gear transmission system.