Though there have been many applications of object tracking, ranging from surveillance and monitoring to smart rooms, object tracking is always a challenging problem in computer vision over the past decades. Mean Shift-based object tracking has received much attention because it has a great number of advantages over other object tracking algorithms, e.g. real time, robust and easy to implement. In this survey, we first introduce the basic principle of the Mean Shift algorithm and the working procedure using the Mean Shift algorithm to track the object. This paper then describes the defects and potential issues of the traditional Mean Shift algorithm. Finally, we summarize the improvements to the Mean Shift algorithm and some hybrid tracking algorithms that researchers have proposed. The main improvements include scale adaptation, kernel selection, on-line model updating, feature selection and mode optimization, etc.