Feature selection and fusion is of crucial importance in multi‐feature visual tracking. This study proposes a multi‐task kernel‐based sparse learning method for multi‐feature visual tracking. The proposed sparse learning method can discriminate the reliable and unreliable features for optimal multi‐feature fusion through using a Fisher discrimination criterion‐based multi‐objective model to adaptively train the kernel weights of different features such as pixel intensity, edge and texture. To guarantee a robustness of the sparse representation method, a mixed norm is employed in the sparse leaning method to adaptively select correlated particle observations for multi‐task sparse reconstruction. Experimental results show that the proposed sparse learning method can achieve a better tracking performance than state‐of‐the‐art tracking methods do.