Modality-Adaptive Mixup and Invariant Decomposition for RGB-Infrared Person Re-identification

Huang, Zhipeng; Liu, Jiawei; Li, Liang; Zheng, Kecheng; Zha, Zheng-Jun

doi:10.1609/aaai.v36i1.19987

Cited by 60 publications

(3 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, a comparison between the proposed method and state-of-the-art VI Re-ID approaches is conducted, including Cross-modal feature learning: DDAG [ 21 ], AGW [ 4 ], cm-SSFT [ 45 ], GLMC [ 22 ] (best method), MPANet [ 12 ], LBA [ 46 ], CM-NAS [ 19 ], MMN [ 47 ], MID [ 48 ]; Modal transformation: cmGAN [ 13 ], AlignGAN [ 49 ], Xmodal [ 17 ], SFANet [ 50 ], AGMNet [ 51 ] (best method), and PMT [ 25 ]. According to Table 1 , the results on the two datasets demonstrate that the proposed GSMEN outperforms state-of-the-art methods, achieving the outstanding performance.…”

Section: Methodsmentioning

confidence: 99%

Graph Sampling-Based Multi-Stream Enhancement Network for Visible-Infrared Person Re-Identification

Jiang,

Xiao,

Wang

et al. 2023

Sensors

View full text Add to dashboard Cite

With the increasing demand for person re-identification (Re-ID) tasks, the need for all-day retrieval has become an inevitable trend. Nevertheless, single-modal Re-ID is no longer sufficient to meet this requirement, making Multi-Modal Data crucial in Re-ID. Consequently, a Visible-Infrared Person Re-Identification (VI Re-ID) task is proposed, which aims to match pairs of person images from the visible and infrared modalities. The significant modality discrepancy between the modalities poses a major challenge. Existing VI Re-ID methods focus on cross-modal feature learning and modal transformation to alleviate the discrepancy but overlook the impact of person contour information. Contours exhibit modality invariance, which is vital for learning effective identity representations and cross-modal matching. In addition, due to the low intra-modal diversity in the visible modality, it is difficult to distinguish the boundaries between some hard samples. To address these issues, we propose the Graph Sampling-based Multi-stream Enhancement Network (GSMEN). Firstly, the Contour Expansion Module (CEM) incorporates the contour information of a person into the original samples, further reducing the modality discrepancy and leading to improved matching stability between image pairs of different modalities. Additionally, to better distinguish cross-modal hard sample pairs during the training process, an innovative Cross-modality Graph Sampler (CGS) is designed for sample selection before training. The CGS calculates the feature distance between samples from different modalities and groups similar samples into the same batch during the training process, effectively exploring the boundary relationships between hard classes in the cross-modal setting. Some experiments conducted on the SYSU-MM01 and RegDB datasets demonstrate the superiority of our proposed method. Specifically, in the VIS→IR task, the experimental results on the RegDB dataset achieve 93.69% for Rank-1 and 92.56% for mAP.

show abstract

Section: Methodsmentioning

confidence: 99%

Graph Sampling-Based Multi-Stream Enhancement Network for Visible-Infrared Person Re-Identification

Jiang,

Xiao,

Wang

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

“…In order to alleviate the problem that single-modal visible light data cannot provide useful information at night (Zhu et al 2021;Rao et al 2021;Wang et al 2022a;Li, Wu, and Zheng 2021;Zheng et al 2015;Zhou et al 2023;Li et al 2022;Lei et al 2008), and eliminate the huge domain heterogeneity between cross-modal data (Zhang et al 2022;Tian et al 2021;Wu et al 2021;Chen et al 2021;Wu et al 2017;Nguyen et al 2017;Huang et al 2022;Farooq et al 2022;Kim et al 2023;Feng, Wu, and Zheng 2023;Zhang and Wang 2023;Zheng et al 2023;He et al 2015), Zheng et al (Zheng et al 2021) construct the first multi-modal person ReID dataset, RGBNT201. And Zheng et al (Li et al 2020) propose the benchmark multi-modal vehicle ReID datasets, RGBN300 and RGBNT100.…”

Section: Related Workmentioning

confidence: 99%

Heterogeneous Test-Time Training for Multi-Modal Person Re-identification

Wang,

Huang,

Zheng

et al. 2024

AAAI

View full text Add to dashboard Cite

Multi-modal person re-identification (ReID) seeks to mitigate challenging lighting conditions by incorporating diverse modalities. Most existing multi-modal ReID methods concentrate on leveraging complementary multi-modal information via fusion or interaction. However, the relationships among heterogeneous modalities and the domain traits of unlabeled test data are rarely explored. In this paper, we propose a Heterogeneous Test-time Training (HTT) framework for multi-modal person ReID. We first propose a Cross-identity Inter-modal Margin (CIM) loss to amplify the differentiation among distinct identity samples. Moreover, we design a Multi-modal Test-time Training (MTT) strategy to enhance the generalization of the model by leveraging the relationships in the heterogeneous modalities and the information existing in the test data. Specifically, in the training stage, we utilize the CIM loss to further enlarge the distance between anchor and negative by forcing the inter-modal distance to maintain the margin, resulting in an enhancement of the discriminative capacity of the ultimate descriptor. Subsequently, since the test data contains characteristics of the target domain, we adapt the MTT strategy to optimize the network before the inference by using self-supervised tasks designed based on relationships among modalities. Experimental results on benchmark multi-modal ReID datasets RGBNT201, Market1501-MM, RGBN300, and RGBNT100 validate the effectiveness of the proposed method. The codes can be found at https://github.com/ziwang1121/HTT.

show abstract

“…Inter modality mixing: [22] learns a dynamical and local linear interpolation between the different regions of cross-modality images in data-dependent fashion to mix up the RGB and infrared (IR) images. We explored both the static and dynamic mixing methods and found the static has a better performance.…”

Section: Related Workmentioning

confidence: 99%

You Only Need One Detector: Unified Object Detector for Different Modalities based on Vision Transformers

Shen¹,

Li²,

Canizales³

et al. 2022

Preprint

View full text Add to dashboard Cite

Most systems use different models for different modalities, such as one model for processing RGB images and one for depth images. Meanwhile, some recent works discovered that an identical model for one modality can be used for another modality with the help of cross modality transfer learning. In this article, we further find out that by using a vision transformer together with cross/inter modality transfer learning, a unified detector can achieve better performances when using different modalities as inputs. The unified model is useful as we don't need to maintain separate models or weights for robotics, hence, it is more efficient. One application scenario of our unified system for robotics can be: without any model architecture and model weights updating, robotics can switch smoothly on using RGB camera or both RGB and Depth Sensor during the day time and Depth sensor during the night time . Experiments on SUN RGB-D dataset show: Our unified model is not only efficient, but also has a similar or better performance in terms of mAP50 based on SUNRGBD16 category: compare with the RGB only one, ours is slightly worse (52.3 → 51.9). compare with the point cloud only one, we have similar performance (52.7 → 52.8); When using the novel inter modality mixing method proposed in this work, our model can achieve a significantly better performance with 3.1 (52.7 → 55.8) absolute improvement comparing with the previous best result. Code (including training/inference logs and model checkpoints) is available: https://github.com/liketheflower/YONOD.git

show abstract

Modality-Adaptive Mixup and Invariant Decomposition for RGB-Infrared Person Re-identification

Cited by 60 publications

References 29 publications

Graph Sampling-Based Multi-Stream Enhancement Network for Visible-Infrared Person Re-Identification

Graph Sampling-Based Multi-Stream Enhancement Network for Visible-Infrared Person Re-Identification

Heterogeneous Test-Time Training for Multi-Modal Person Re-identification

You Only Need One Detector: Unified Object Detector for Different Modalities based on Vision Transformers

Contact Info

Product

Resources

About