The gear transmission system is one of the essential subsystems of railway vehicles. The wheel polygonalization can affect the gear transmission system’s condition monitoring and fault diagnosis, causing fatigue failure of the gear system and influencing driving safety. Therefore, to explore the mechanism of the wheel polygonalization in the gear system, this paper develops a Lagrangian form dynamic model of a railway vehicle with a gear transmission system. The model validation uses the data from the field test. The analysis explores the influence of the wheel polygonal fault on the gear transmission system under the traction condition. The results indicate that the wheel polygonal fault has amplitude/frequency modulation effects on the time/frequency response of the gear system. Also, the wheel polygonal fault causes the envelope of the speed curve to fluctuate periodically at a low frequency under traction conditions. The waveform asymmetry is gradually apparent as the traction continues. In the non-resonant region, the statistical analysis of the rotational speed fluctuation illustrates that the order of the wheel polygonal fault has more impact on the rotational speed fluctuation than the amplitude. Besides, in the resonance region, the eigenvalues of the system determine the magnitude of the amplitude.