Periodic and period-doubling vibrations in a gear transmission system subjected to forced excitations with multi-piecewise linear functions are investigated in this work by using the incremental harmonic balance (IHB) method. The nonlinear ordinary differential equations that govern vibration of the gear transmission system are formulated by employing the Newton's second law. Analytical results reveal abundant interesting phenomena, including jumps, bifurcations, softening-spring behaviors, and primary, super-harmonic, and sub-harmonic resonances, which have not been exhibited in existing investigations on nonlinear vibrations of gear transmission systems. Nonlinear phenomena and resonances of the gear transmission system are revealed by considering different numbers of degrees of freedom of the system. The bifurcations containing saddle-node, period-doubling, and Hopf bifurcations are observed in frequency response curves. The period-doubling phenomena are characterized with phase-plane diagrams and Fourier spectra. Further, analytical results obtained by the IHB method match very well with those from numerical integration.