Vibration is a common and urgent technical issue in the steel industry. The world’s first multi-mode continuous-casting and rolling plant of Shou Gang Jing Tang Iron and Steel Co., Ltd. (Tangshan, China), has a finishing mill, F3, that experiences frequent, strong vibrations during the process of rolling thin-gauge (<1.5 mm) strip steel, which have seriously hindered the production of high-quality thin strip steel products. The changes in the strips’ surface quality are among the factors that induce rolling mill vibrations. In this study, considering the nonlinear surface quality of strip steel, a finite element model of the F3 mill was established, and the harmonic response method was used to obtain a rolling mill vertical system in the ANSYS environment. This study assesses the sensitive amplitude versus frequency characteristics curve of a torsional coupling system, the influence of strip thickness and strip hardness fluctuations on the vibration of the primary drive system, and the dynamic amplitude versus frequency characteristics of the three directions on the top of a torii. Finally, the field experiment verifies the correctness of the analytical results, which provides theoretical guidance for suppressing rolling mill vibrations and has a certain application value.