In this paper, we analyze the key problems in the tracking of the specified target on the complex infrared background, for example how to judge the polarity of target, how to select the seed of region growing, and how to determine the suitable size of the detection window. We propose a reasonable solution for each problem, such as to judge the target polarity by comparing the gray value, and to segment the target by using the adaptive region growing and then determine the size of detection window. In this p銆乤per, we propose the spokes variance (SV) method to describe the shape, and use the features of target to exclude interference, which include area, location, gray value and shape feature. Through the Kalman Filter to predict position of target, and solve the problem of target being covered in short time. The experimental results show that the algorithm is simple and effective, and achieved good tracking effect.