Robust Multi-Modality Multi-Object Tracking

Zhang, Wenwei; Zhou, Hui; Sun, Shuyang; Wang, Zhe; Shi, Jianping; Loy, Chen Change

doi:10.1109/iccv.2019.00245

Cited by 202 publications

(131 citation statements)

References 42 publications

Supporting

Mentioning

131

Contrasting

Order By: Relevance

“…To demonstrate the effectiveness of our online multi-object tracking method, we compared our algorithm to several state-of-the-art approaches using both the KITTI and ATTD, including offline tracking methods like Siamese CNN [47], Convolutional Neural Networks and Temporally Constrained Metrics (CNNTCM) [48], Discrete-Continuous Energy Minimization (DCO-X) [49], and Learning Optimal Structured Support Vector Machine LP-SSVM [50], Online tracking methods like Near-Online Multi-Target Tracking (NOMT-HM) [51], Structural Constraint Event Aggregation (SCEA) [52], Spatial-Temporal Attention Mechanism (STAM) [3], Successive Shortest Path (SSP) [53], multi-modality Multi-Object Tracking (mmMOT) [54], and Multi-Object Tracking Beyond Pixels (MOTBeyondPixels) [55]. Some of these approaches could only be performed in the offline setting.…”

Section: Benchmark Evaluation Resultsmentioning

confidence: 99%

Spatial–Semantic and Temporal Attention Mechanism-Based Online Multi-Object Tracking

Meng

Wang

et al. 2020

Sensors

View full text Add to dashboard Cite

Multi-object tracking (MOT) plays a crucial role in various platforms. Occlusion and insertion among targets, complex backgrounds and higher real-time requirements increase the difficulty of MOT problems. Most state-of-the-art MOT approaches adopt the tracking-by-detection strategy, which relies on compute-intensive sliding windows or anchoring schemes to detect matching targets or candidates in each frame. In this work, we introduce a more efficient and effective spatial–temporal attention scheme to track multiple objects in various scenarios. Using a semantic-feature-based spatial attention mechanism and a novel Motion Model, we address the insertion and location of candidates. Some online-learned target-specific convolutional neural networks (CNNs) were used to estimate target occlusion and classify by adapting the appearance model. A temporal attention mechanism was adopted to update the online module by balancing current and history frames. Extensive experiments were performed on Karlsruhe Institute of Technologyand Toyota Technological Institute (KITTI) benchmarks and an Armored Target Tracking Dataset (ATTD) built for ground-armored targets. Experimental results show that the proposed method achieved outstanding tracking performance and met the actual application requirements.

show abstract

Section: Benchmark Evaluation Resultsmentioning

confidence: 99%

Spatial–Semantic and Temporal Attention Mechanism-Based Online Multi-Object Tracking

Meng

Wang

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Learning is driven by reconstruction error based on backpropagation. Zhang et al [27] tracked objects in multi-modal scenarios by adopting a deep architecture that can be trained in an end-to-end manner, thereby enabling the joint optimization of the base feature extractors of each modality and an adjacency estimator for cross-modality. Wen et al [28] proposed an MOT algorithm based on a non-uniform hypergraph that can model different degrees of dependency among tracklets for a unified objective.…”

Section: Related Workmentioning

confidence: 99%

OneShotDA: Online Multi-Object Tracker With One-Shot-Learning-Based Data Association

et al. 2020

View full text Add to dashboard Cite

Tracking multiple objects in a video sequence can be accomplished by identifying the objects appearing in the sequence and distinguishing between them. Therefore, many recent multi-object tracking (MOT) methods have utilized re-identification and distance metric learning to distinguish between objects by computing the similarity/dissimilarity scores. However, it is difficult to generalize such approaches for arbitrary video sequences, because some important information, such as the number of objects (classes) in a video, is not known in advance. Therefore, in this study, we applied a one-shot learning framework to the MOT problem. Our algorithm tracks objects by classifying newly observed objects into existing tracks, irrespective of the number of objects appearing in a video frame. The proposed method, called OneShotDA, exploits the one-shot learning framework based on an attention mechanism. Our neural network learns to classify unseen data samples using labels from a support set. Once the network has been trained, it predicts correct labels for newly received detection results based on the set of existing tracks. To analyze the effectiveness of our method, it was tested on the MOTchallenge benchmark datasets (MOT16 and MOT17 datasets). The results reveal that the performance of the proposed method was comparable with those of current state-of-the-art methods. In particular, it is noteworthy that the proposed method ranked first among the online trackers on the MOT17 benchmark.

show abstract

“…Multimodal machine learning aims to build models that can process, correlate, and integrate information from multiple modalities [2]. The success of multimodal machine learning has been demonstrated in a wide range of applications, e,g, human action analysis [1,4,37,38] , person/object localization and tracking [15,34,47] and image segmentation [14,51].…”

Section: Multimodal Machine Learningmentioning

confidence: 99%

Adaptive Multimodal Fusion for Facial Action Units Recognition

Yang

Wang

Liu

2020

Proceedings of the 28th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Multimodal facial action units (AU) recognition aims to build models that are capable of processing, correlating, and integrating information from multiple modalities (i.e., 2D images from a visual sensor, 3D geometry from 3D imaging, and thermal images from an infrared sensor). Although the multimodel data can provide rich information, there are two challenges that have to be addressed when learning from multimodal data: 1) the model must capture the complex cross-modal interactions in order to utilize the additional and mutual information effectively; 2) the model must be robust enough in the circumstance of unexpected data corruptions during testing, in case of a certain modality missing or being noisy. In this paper, we propose a novel Adaptive Multimodal Fusion method (AMF) for AU detection, which learns to select the most relevant feature representations from different modalities by a re-sampling procedure conditioned on a feature scoring module. The feature scoring module is designed to allow for evaluating the quality of features learned from multiple modalities. As a result, AMF is able to adaptively select more discriminative features, thus increasing the robustness to missing or corrupted modalities. In addition, to alleviate the over-fitting problem and make the model generalize better on the testing data, a cut-switch multimodal data augmentation method is designed, by which a random block is cut and switched across multiple modalities. We have conducted a thorough investigation on two public multimodal AU datasets, BP4D and BP4D+, and the results demonstrate the effectiveness of the proposed method. Ablation studies on various circumstances also show that our method remains robust to missing or noisy modalities during tests. CCS CONCEPTS • Computing methodologies → Activity recognition and understanding; Biometrics; Image representations.

show abstract

Robust Multi-Modality Multi-Object Tracking

Cited by 202 publications

References 42 publications

Spatial–Semantic and Temporal Attention Mechanism-Based Online Multi-Object Tracking

Spatial–Semantic and Temporal Attention Mechanism-Based Online Multi-Object Tracking

OneShotDA: Online Multi-Object Tracker With One-Shot-Learning-Based Data Association

Adaptive Multimodal Fusion for Facial Action Units Recognition

Contact Info

Product

Resources

About