Abstract. In the traditional background subtraction algorithms based on color, in the special critical situation, such as face the challenges of shadows, color camouflage, illumination changes, they can't achieve good results. But depth data can solve such problems effectively, because depth sensor operating normally in the scene. We proposed an improved mixture of Gaussian model algorithm based on color and depth data in this paper, and use a new decision-making method to fuse results. In order to ensure the stability of the model, we have special treatment of the ADO area. We test our method in three datasets, the results show that proposed approach obtain good results.