Active magnetic bearing (AMB) is competent in rotor trajectory control for potential applications such as mechanical processing and spindle attitude control, while the highly nonlinear and coupled dynamic characteristics especially in the condition of rotor large motion are obstacles in controller design. In this paper, a controller of AMB is proposed to achieve rotor 3D trajectory control. First, the dynamic model of the AMB-rotor system containing a nonlinear electromagnetic force model is introduced. Then the DCNN-SMC (deep convolutional neural network - sliding mode control) controller is proposed. Sliding mode control is used to achieve the tracking control with high robustness and responsiveness, and a deep convolutional neural network based on deep learning method is designed to compensate the uncertainties of the system. Finally, simulation of a 5-degree of freedom (DOF) system on various trajectories demonstrates evident control effect of the proposed controller in precision and significant effect of DCNN based on deep learning method in compensation control.