The helical gear pair of a box-type vibration exciter of a mine-used linear vibrating screen is subjected to complex excitation and prone to broken tooth failures. At present, investigation regarding the difference and particularity between gear transmission in vibrating screens (i.e., vibration machinery) and that in rotating machinery is still a challenge, which is the key to revealing the performance and failure mechanism of gear transmission in the premise of application to vibrating screens. In order to intuitively display the peculiarity of gear transmission on the exciter, an innovative virtual prototype model of a gear pair of a vibrating screen exciter is proposed. This model considers the effects of internal and external excitation, such as the friction and lubrication of the gear, strong alternating load produced by a large eccentric block, the reciprocating motion of the screen body, the large clearance of bearing and so on, and its correctness is verified. Based on the comparison, the inducement for the high fatigue rate of exciter gears is revealed. Models of a vibrating screen’s excitation system with different degrees of broken teeth are also established, and tooth fault features are proposed for fault detection. Sensitive indicators for the degradation degree of tooth damage are put forward, and the monitoring strategy is presented that with the increase of damage degree, the waveform index and pulse index of axial vibration acceleration increase. The analysis results provide powerful support for the optimal design of the vibrating screen’s exciter gears and fault diagnosis.