The mode transition of single-shaft parallel hybrid electric vehicles (HEVs) between engine and motor has an important impact on power and drivability. Especially, in the process of mode transition from the pure motor-drive operating mode to the only engine-drive operating mode, the motor starting engine and the clutch control problem have an important influence on driving quality, and solutions have a bit of room for improving dynamic performance. In this paper, a novel mode transition control method is proposed to guarantee a fast and smooth mode transition process in this regard. First, an adaptive sliding mode control (A-SMC) strategy is presented to obtain the desired torque trajectory of the clutch transmission. Second, a proportional-integral (PI) observer is designed to estimate the actual transmission torque of the clutch. Meanwhile, a fractional order proportional-integral-differential (FOPID) controller with the optimized control parameters by particle swarm optimization (PSO) is employed to realize the accurate position tracking of the direct current (DC) motor clutch so as to ensure clutch transmission torque tracking. Finally, the effectiveness and adaptability to system parameter perturbation of the proposed control approach are verified by comparison with the traditional control strategy in a MATLAB environment. The simulation results show that the driving quality of the closed-loop system using the proposed control approach is obviously improved due to fast and smooth mode transition process and better adaptability.