The rotor is an essential actuator of quadrotor UAV, and is prone to failure due to high speed rotation and environmental disturbances. It is difficult to diagnose rotor faults and identify the fault localization simultaneously. In this paper, we propose a fault diagnosis and localization scheme based on the Extended State Observer (ESO) and Deep Forest (DF). This scheme can accurately complete the fault diagnosis and localization for the quadrotor UAV actuator without knowing the fault size by combining the model-based and the data-driven methods. First, we obtain the angular acceleration residual signal of the quadrotor UAV by using ESO. The residual signal is the difference between the observed state of ESO and the true fault state. Then, we design the residual feature analysis method by considering the position distribution of the quadrotor UAV actuator. This method can embed the actuator fault localization information into the fault data by simultaneously considering pitch and roll of the quadrotor UAV. Finally, we complete the fault diagnosis and localization of the quadrotor UAV actuator by processing the fault data by using DF. This scheme has the advantages of straightforward observer modeling, strong generalization ability, adaptability to small sample data, and few hyperparameters. Our simulation results indicate that the accuracy of the proposed scheme reaches more than 99% for the unknown size of the quadrotor UAV actuator fault.