The research on the nonlinear vibration compensation method in the field of forging hydraulic press is still in the blank stage. This paper studies the nonlinear vibration compensation problem of forging hydraulic press under the action of a kind of nonlinear external system and realizes the increase of a kind of nonlinear vibration in the nonlinear system of a hydraulic press. This paper proposes two methods to solve the nonlinear relationship. One is to concatenate an inverse function of a nonlinear relationship and synthesize it to obtain a linear relationship. While another is to acknowledge its nonlinear relationship and determine its output based on its nonlinear relationship. Furthermore, this paper chooses the tabu search algorithm to solve the compensation problem of nonlinear vibration and improves its search neighborhood to solve the problem of vibration range selection, analyses the vibration reinforcement calculation, and describes the specific steps of applying the tabu search algorithm. Finally, MATLAB is used to simulate and validate the designed method. The simulation results show that the method can suppress nonlinear vibration and the corresponding compensation effectiveness, proving the design method’s rationality.