The capability of recognizing and tracking a specific human is considered a key technique for serving mobile robots. This paper proposes a real‐time tracker that uses a stereo camera to track specific people in different environments. A tracker has been designed to detect the selected target from multiple people by use of block matching algorithm, human detection, and colour histogram comparison. Considering the similar colour of the objects, the predictor of the Kalman filter determines the position of the object. Finally, the mobile robot is moved according to the tracking results with a relative distance between the robot and the target. The effectiveness of the proposed method is demonstrated by experiments on many video sequences and real environments.