In this paper, the vibration suppression and trajectory tracking control of a flexible joint manipulator (FJM) based on particle swarm optimization (PSO) and fixed-time nonsingular terminal sliding mode control (NTSMC) are studied. Firstly, in order to suppress the residual vibration of the FJM, an optimal trajectory planning method based on higher-order trajectory planning (HOTP) and the PSO algorithm is proposed. Then, to ensure that the FJM can track the optimized trajectory without being affected by the initial value of the trajectory, a novel fixed-time NTSMC scheme is proposed. Compared with the cubic spline trajectory, the proposed HOTP is smoother and can more accurately suppress the residual vibration of the FJM. By combining the HOTP with the PSO algorithm, the vibration amplitude of FJM can be suppressed to around 0.002 mm. Unlike finite-time NTSMC, the rate of convergence of the proposed fixed-time NTSMC does not depend on the initial value of FJM’s joint trajectory. Especially when the initial value of the trajectory is large, the FJM can still quickly track the optimal trajectory within 0 to 1 s. Finally, the effectiveness of this method is verified through simulation and comparison.