This paper presents the trajectory tracking control of a flexible-joint manipulator driven by permanent magnet synchronous motor (PMSM). Combining the PMSM electrical equation and mechanical equation of robotic manipulators, a novel smooth switching control scheme is proposed. Firstly, the position loop controller of the system is designed, with an improved hierarchical sliding mode control (IHSMC) algorithm proposed to further the response speed of the system, additionally, a robust interconnection and damping assignment passivity-based controller (IDA-PBC) is designed to improve the steady state performance of the system. Then, the IDA-PBC control strategy is leveraged to design the current loop controller of the system, on which basis a hybrid controller with smooth switching is designed. Furthermore, Gaussian function is applied as the smooth switching function of the hybrid controller to promote the switching performance. As a result, the hybrid controller has both good dynamic and steady performance. The simulation results verify the effectiveness of the algorithms.