PurposeGear transmissions are widely utilized in practice. This paper aims to uncouple the crack feature from the cracked time-varying mesh stiffness (TVMS) and investigate the effects of the crack on the nonlinear dynamics of a spur gear pair.Design/methodology/approachAn approximate method to simulate the cracked TVMS is proposed by using an amplitude modulation function. The ratio of mesh stiffness loss is introduced to estimate the TVMS with different crack depths and angles. The dynamic responses are obtained by solving a torsional model which takes the non-loaded static transmission error, the backlash and the cracked TVMS into account. By using the bifurcation diagram, the largest Lyapunov exponent (LLE) and dynamic mesh force, the influences of crack on nonlinear behaviors are examined. The dynamic characteristics are identified from the phase diagram, Poincaré map, dynamic mesh force, time series and FFT spectra.FindingsThe comparison between the healthy and cracked gear pairs indicates that the crack affects the system motions, such as the obvious changes of impact force and unpredictable instability. Besides, the additive and difference combination frequencies can be found in periodic-1 and -2 motions, but they are covered in periodic-3 and chaotic motions. Deeper crack is an important determinant of the nonlinear behaviors at a higher speed.Originality/valueThe research provides an interesting perspective on cracked TVMS and reveals the connection between crack and nonlinear behaviors of the gear pairs.