Video target tracking has a wide range of application value in the field of automatic driving, UAV target tracking, security monitoring, etc. How to maintain stable tracking of the target among video data frames is the focus of the research. A robust tracking algorithm that effectively solves the target drift problem is proposed for the problem of target loss due to image perturbation, scale change, target occlusion and other disturbances when the KCF algorithm is used for video target tracking. The algorithm is based on the KCF algorithm framework, which proposes a multi-scale sampling strategy and designs a multiple classifier screening algorithm to ensure the accuracy of the target template. Through experimental verification, the algorithm can effectively solve the drift problem in the tracking process and realize the continuous accurate tracking of the target. The algorithm provides a real-world application reference for engineering applications of real-time video data processing.