In the present study, the chaotic response of the nonlinear magnetostrictive actuator (GMA) vibration system is investigated. e mathematical model of the nonlinear GMA vibration system is established according to J-A hysteresis nonlinear model, quadratic domain rotation model, Newton's third law, and principle of GMA structural dynamics by analyzing the working principle of GMA. en, the Melnikov function method is applied to the threshold condition of the chaotic response of the system to obtain the sense of Smale horseshoe transformation. Furthermore, the mathematical model is solved to investigate the system response to the excitation force and frequency. Accordingly, the corresponding displacement waveform, phase plane trajectory, Poincaré map, and amplitude spectrum are obtained. e experimental simulation is verified using Adams software. e obtained results show that the vibration equation of the nonlinear GMA vibration system has nonlinear and complex motion characteristics with different motion patterns. It is found that the vibration characteristics of the system can be controlled through adjusting the excitation force and frequency.