The improved mean shift (IMS) algorithm can effectively track a colour target undergoing fast motion and complete occlusion. Instead template matching based on the sum of weighted absolute differences (SWAD) can track a target precisely, even under partial occlusion. To take advantage of both trackers, this paper presents a novel tracking algorithm that, on one side, further improves the IMS and extends it to a more general and flexible framework; on the other side, the proposed algorithm incorporates SWAD minimisation to increase the precision of the tracking results. Experimental results demonstrate that such an algorithm can robustly and precisely track targets with fast motion and in complete occlusion, more than conventional mean shift, IMS and SWAD-based trackers taken separately.