The fluid–structure interaction effect should not be disregarded when examining the vibration characteristics of hydraulic pipeline systems. The transfer matrix method (TMM) is an efficacious method for analyzing the vibration characteristics of hydraulic pipelines in the frequency domain, offering advantages such as simplicity and efficiency. However, the TMM suffers the problem of high frequency instability when dealing with long-span hydraulic pipelines, which restricts its practical application. Currently, several modified transfer matrix methods face challenges such as low computational efficiency and difficulties in handling complex boundaries. In response to these issues, this paper proposes a novel modified transfer matrix method known as the mixed variable transfer matrix method. This innovative method possesses clear physical significance and effectively prevents the transfer matrix from becoming singular without necessitating the subdivision of the pipeline length. Consequently, it addresses high-frequency instability while maintaining high computational efficiency. Moreover, this method is capable of addressing complex boundary problems by integrating boundary matrices, thereby demonstrating enhanced applicability compared to existing methods. The performance of the proposed method was validated through the utilization of classic Dubee pipeline impact test data, and the result shows maximum errors of 3.03% relative to the public data. Subsequently, an experiment was conducted on a section of hydraulic piping within a ship’s steering system. A hydraulic fluid noise generator was established to induce fluid pulsation excitation to the pipeline, thereby simulating the actual boundary conditions encountered in a ship’s hydraulic pipeline system so as to corroborate the efficacy of the proposed method in predicting the frequency domain vibration characteristics of a real hydraulic pipeline system. The experimental results indicate that the proposed method offers significant advantages in terms of high precision, efficiency, and stability, shows maximum errors of 4.35% relative to experimental data, and demonstrates promising prospects for engineering applications.