Robot Person Following in Uniform Crowd Environment

Abstract

Person-tracking robots have many applications, such as in security, elderly care, and socializing robots. Such a task is particularly challenging when the person is moving in a Uniform crowd. Also, despite significant progress of trackers reported in the literature, state-of-the-art trackers have hardly addressed person following in such scenarios. In this work, we focus on improving the perceptivity of a robot for a person following task by developing a robust and real-time applicable object tracker. We present a new robot person tracking system with a new RGB-D tracker, Deep Tracking with RGB-D (DTRD) that is resilient to tricky challenges introduced by the uniform crowd environment. Our tracker utilizes transformer encoder-decoder architecture with RGB and depth information to discriminate the target person from similar distractors. A substantial amount of comprehensive experiments and results demonstrate that our tracker has higher performance in two quantitative evaluation metrics and confirms its superiority over other SOTA trackers.• We propose a fully autonomous robot person following system with our new RGB-D tracker. It can track the person under the challenges like uniform crowd and occlusion with similar-appearing people.

Lightweight Multimodal Domain Generic Person Reidentification Metric for Person-Following Robots

“…Once the bounding box of target person predicted, the state of Kalman filter is updated by identifying the target bounding box, and the target is then followed. In [ 3 ], another problem of tracking the target in uniform crowd environment is solved. Their method depends on accurate face identification because the target and other persons all have similar appearances in the scene.…”

“…[ 14 ] can be distracted due to impostors, whereas Ref. [ 3 ] can be distracted in crowded environment due to poor depth sensing.…”

“…In Figure 1 , it can also be noted that the real world is also crowded where the target person P 1 is also either occluded by other persons, say occluded by distractor D 1 at time t 2 on walking across 1, or is also occluded by impostor I 1 at time t 4. Thus, in the real world in Figure 1 , due to occlusion and nearby impostors, tracking the target person P 1 is a very difficult task, hence, state-of-the-art trackers [ 3 , 4 , 5 , 6 ] could lose the tracking of target person P 1, and either wrongly start following distractor D 1 at time t 4, or wrongly start following an impostor person I 1 at time t 5, as shown in red rectangles in Figure 1 . The real target person P 1 is walking on 1 at time t 4, and on 4 at time t 5, respectively.…”

“…Therefore, to overcome these shortcomings in visual trackers [ 11 ], recently, reidentification metrics have been learned and integrated with visual trackers [ 12 ] to follow the target person [ 1 , 2 ]. These reidentification metrics are learned by matching color-histograms and gait features [ 1 , 13 ], as well as extracting deep CNN features to learn deep similarity metrics [ 2 , 3 , 14 , 15 , 16 ]. However, the reidentification metrics in the present works [ 2 , 3 , 14 , 15 , 16 ] are all learned assuming the naïve world, i.e., it is assumed that the outside world is close-set, unimodal (it is assumed the robot only uses RGB sensor), and the target person P 1 appearance remains unchanged across moving different domains.…”

“…These reidentification metrics are learned by matching color-histograms and gait features [ 1 , 13 ], as well as extracting deep CNN features to learn deep similarity metrics [ 2 , 3 , 14 , 15 , 16 ]. However, the reidentification metrics in the present works [ 2 , 3 , 14 , 15 , 16 ] are all learned assuming the naïve world, i.e., it is assumed that the outside world is close-set, unimodal (it is assumed the robot only uses RGB sensor), and the target person P 1 appearance remains unchanged across moving different domains. Due to these naïve assumptions, not only do the generalization capabilities of the learned reidentification metrics suffer largely in tracking the target in the outside world, but these metrics also get distracted due to the impostors observed across complex scenes and across nonlinear domains during tracking target P 1, as shown in Figure 1 .…”