For the helical gear fault diagnosis investigation, it is essential to establish a pitting dynamics model that incorporates several nonlinear parameters to simulate the fault characteristics. This paper proposes a multi-degree-of-freedom analytical nonlinear dynamic model of helical gear pair under pitting fault, introducing the stiffness and damping parameters of gear matrix and lubricant film, the variable friction coefficient of elastohydrodynamic lubrication (EHL), and the measured meshing profile error. Contact fatigue experiments are carried out with a helical gear transmission test bench to gain the full-stage vibration response from healthy to severe pitting fault conditions to validate the reliability of the proposed nonlinear dynamics model. The distribution and amplitude of the eigenfrequencies of the analytical results, considering the influence of the lubricating oil film, are more consistent with the experimental measurements, where the sidebands between the three times and four times fundamental frequency are in acceptable agreement. The comparison results confirm the importance of considering the influence of the lubricant film when studying the dynamics of gear tooth pitting failures, and the analytical method and proposed model can be a reference for dynamical simulation modelling for gear fatigue failure diagnosis and analysis.