In order to solve the problem that the conventional tracker is not adapted to the abrupt motion, a tracking algorithm based on the improved salp swarm algorithm (ISSA) was proposed. Visual tracking is considered to be a process of locating the optimal position through the interaction between leaders and followers in successive images. Firstly, the adaptive mechanism of leader and follower is introduced into the original salp swarm algorithm (SSA) to balance the exploitation and exploration of the algorithm. This method can improve the accuracy and effect of tracking. Secondly, the golden‐sine algorithm was used to update the position of followers, considering that the SSA had a single spatial search mode for followers and was easy to fall into the local optimum. By comparing with 19 classical tracking algorithms, qualitative and quantitative analysis is carried out to verify the tracking effect of the proposed method. A large number of experimental results show that the algorithm proposed here has good performance in visual tracking, especially for mutation motion tracking.