Synthetic Data Are as Good as the Real for Association Knowledge Learning in Multi-object Tracking

Abstract

Association, aiming to link bounding boxes of the same identity in a video sequence, is a central component in multi-object tracking (MOT). To train association modules, e.g., parametric networks, real video data are usually used. However, annotating person tracks in consecutive video frames is expensive, and such real data, due to its inflexibility, offer us limited opportunities to evaluate the system performance w.r.t changing tracking scenarios. In this paper, we study whether 3D synthetic data can replace real-world videos for association training. Specifically, we introduce a large-scale synthetic data engine named MOTX, where the motion characteristics of cameras and objects are manually configured to be similar to those in real-world datasets. We show that compared with real data, association knowledge obtained from synthetic data can achieve very similar performance on real-world test sets without domain adaption techniques. Our intriguing observation is credited to two factors. First and foremost, 3D engines can well simulate motion factors such as camera movement, camera view and object movement, so that the simulated videos can provide association modules with effective motion features. Second, experimental results show that the appearance domain gap hardly harms the learning of association knowledge. In addition, the strong customization ability of MOTX allows us to quantitatively assess the impact of motion factors on MOT, which brings new insights to the community 1 .

Recent Advances in Embedding Methods for Multi-Object Tracking: A Survey

“…Since fully annotated real video data is hard to collect, many works [229], [253], [254] try to utilize synthetic data for training trackers. The annotations such as box, mask and track ID, can be naturally obtained from synthetic data.…”

“…For example, [229] shows pre-training on virtual data can improve performance. [253] introduces a large-scale synthetic data engine named MOTX to generate synthetic data for MOT training and achieve competitive results with the trackers trained using real data. In addition, [254] shows that better performance can be achieved with synthetic data on some widely known trackers such as Tracktor [15], Track R-CNN [37], Lift T [255], MPNTrack [6], and CenterTrack [12].…”