Abstract. In this paper, we proposed a real-time visual tracking system for delivery robot. The RGB-D data are collected for environmental perception. The critical step is the tracking algorithm we utilized, this tracking algorithm must be robust, which can easily detect and locate the target, and adapt many complicated environment without loss the target. And moreover, this tracking algorithm must meet the high demand of real-time capability to fulfil the task. In the system designing, a FFT based tracking algorithm is used and the experimental results showed that this algorithm is both robust and efficient, thus is very appropriate for the luggage delivery robotic system.