At present, the active control of gear vibration mostly relies on existing algorithms. In order to achieve effective vibration reduction of the gear system, particularly during the vibration process, this paper proposes a multi-channel VSMFxLMS algorithm based on the FxLMS algorithm. This novel approach takes into account the time-varying nature of the vibration signal during gear vibration. Adaptive filter power coefficients are updated in a skip-tongue variable-step manner using momentum factors. Firstly, the paper establishes the dynamics model of the gear system and analyzes the nonlinear dynamic characteristics of the system. It then examines the vibration damping effect of the FxLMS algorithm and analyzes its performance under different gear system motion states, considering different step lengths and momentum factors. Lastly, the proposed VSMFxLMS algorithm is compared with the FxLMS algorithm, highlighting the superiority of the former. Overall, this research highlights the potential of a multi-channel VSMFxLMS algorithm in reducing vibrations in gear systems. The study optimizes the performance of gear systems while using advanced control strategies.