The vibration level is the key index to measure the performance of marine gear transmission device. The compound gear transmission system, which is composed of a helical gear pair and a spur star gear train, is widely used in marine gear transmission device. This paper presents a multi-node dynamic model for a marine compound gear transmission system, investigating the modal properties and dynamic responses of the system. The mesh stiffness and transmission error for each gear pair are determined using the loaded tooth contact analysis (LTCA) model. The theoretical and experimental vibration acceleration responses of the gearbox housing are contrastively analysed. The results show that some vibration modes are very similar to those of a single planetary gear train, but other modes are multi-node and multi-degree of freedom coupling modes. The vibration coupling and transferring phenomenon between two stages can be observed remarkably, but those show obvious differences under different working conditions. The torsional stiffness of the elastic connecting shaft has significant influence on the vibration coupling and transferring phenomenon. The results can provide useful guidance for the low vibration and noise design of the marine compound gear transmission system.