Vehicle detection and tracking are key aspects of intelligent transportation. However, the accuracy of traditional methods for vehicle detection and tracking is severely suppressed by the complexity of road conditions, occlusion, and illumination changes. To solve the problem, this paper initializes the background model based on multiple frames with a fixed interval, and then establishes a moving vehicle detection algorithm based on trust interval. The established algorithm can easily evaluate the background complexity based on regional information. After that, the authors pointed out that the correlation filtering, a classical vehicle tracking algorithm, cannot adapt to the scale changes of vehicles, due to the weak dependence of background information. Hence, a tracking algorithm that adapts to vehicle scale was designed based on background information. Finally, the proposed algorithms were proved feasible for detection and tracking moving vehicles in complex environments. The research provides a good reference for the application of computer vision in moving target detection.