Chaos analysis and control of relative rotation nonlinear dynamic system with Mathieu-Duffing oscillator are investigated. By using Lagrange equation, the dynamics equation of relative rotation system has been established. Melnikov’s method is applied to predict the chaotic behavior of this system. Moreover, the chaotic dynamical behavior can be controlled by adding the Gaussian white noise to the proposed system for the sake of changing chaos state into stable state. Through numerical calculation, the Poincaré map analysis and phase portraits are carried out to confirm main results.