As an important supporting component, metal rubber clamp often generates nonlinear force and brings complex dynamics to pipeline system. However, the nonlinear of metal rubber clamps in pipeline system has been rarely incorporated in the open research. Therefore, based on genetic algorithm and finite element method, a new nonlinear clamp model with four degrees of freedom is proposed by introducing identified nonlinear parameters. The nonlinear parameters are identified by genetic algorithm in which the objective function is the error between experiment and simulation, and the nonlinear clamp-pipeline model is verified by modal test and vibration response experiment under different boundary conditions. The results show the numerical results are basically consistent with the experimental results. Moreover, the effects of clamp nonlinear stiffness coefficient, nonlinear damping coefficient, and excitation amplitude on vibration response of pipeline system are analyzed. The developed model can consider the nonlinear effect of pipeline system, so as to predict the vibration response more accurately.