In this paper, the fault-tolerant tracking control problem for a class of single-link flexible joint manipulator (SFJM) system with uncertainty, fault, nonlinear function, and unmatched disturbance is investigated. An observer-based sliding mode control approach is designed. Concretely, first of all, the SFJM dynamic system with uncertainty, fault, and unmatched disturbance is established. Then, by transforming the system into two subsystems, a novel composite observer is proposed to estimate the fault and disturbance, respectively. Furthermore, a robust sliding mode controller, which contains a third-order sliding mode surface, a continuous control strategy, and a visual estimated fault signal, is constructed. In the control scheme, an adaptive law is also concluded to compensate for the estimation error. Finally, the proposed method is applied to the SFJM system and the simulation results illustrate the effectiveness of the proposed method.