This paper aims to propose a strategy for the flight control of quad-rotors under single rotor failure conditions. The proposed control strategy consists of two stages—fault detection (FD) and fault tolerant control (FTC). A dual observer-based strategy for FD and fault estimation is developed. With the combination of the results from both observers, the decision making in whether a fault actually happened or the observed anomaly was caused by an external disturbance could be distinguished. Following the FD result, a control strategy for normal flight, as well as the abnormal one, is presented. The FTC considers a real-time coordinate transformation scheme to manipulate the target angles for the quad-rotor to follow a prescribed trajectory. When a rotor fault happens, it is going to be detected by the dual observers and then the FTC is activated to stabilize the system such that the trajectory following task can still be fulfilled. Furthermore, in order to achieve robust flight in the presence of external wind perturbation, the sliding mode control (SMC) theory is further integrated. Simulations illustrate the effectiveness and feasibility of the proposed method.