The star herringbone gear transmission system has a high load-carrying capacity, and is widely used in aviation, marine power drives, off-road vehicles, and hybrid electric-drive vehicles. Vibration and noise are the key concerns with this transmission system. The lumped mass method was adopted to establish the dynamic model and equations of this system. The modes of the system were analyzed and classified, and the eigenvalues and their multiplicities were determined. The results showed that the system has four typical vibration modes: (1) a lateral-rotational coupled vibration mode (multiplicity m = 1), (2) star gear compound mode (multiplicity m = N-3, N > 3), (3) center component lateral vibration mode (multiplicity m = 2), and (4) star gear and center gear-coupled mode (multiplicity m = 2). The contribution of this paper lies in the discovery of the coupling vibration modes in the star herringbone gear transmission system and the multiplicities of these modes. This work provides the foundation for further research on vibration suppression for the star herringbone gear transmission system and the theory of planet phasing.