CAIBC: Capturing All-round Information Beyond Color for Text-based Person Retrieval

Wang, Zijie; Zhu, Aichun; Xue, Jingyi; Wan, Xili; Liu, Chao; Li, Yifeng

doi:10.1145/3503161.3548057

Cited by 52 publications

(8 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By inputting the augmented grayscale image into the model, the model is compelled to focus on other information like texture and shape during training. While color information is known to be critical in TBPS (Wu et al 2021;Wang et al 2022c), these empirical results suggest that other information, apart from color, is also valuable for person retrieval. Conversely, GaussianBlur, which blurs fine-grained details, significantly degrades performance.…”

Section: Ablations Of Data Augmentationmentioning

confidence: 79%

An Empirical Study of CLIP for Text-Based Person Search

Cao,

Bai,

Zeng

et al. 2024

AAAI

View full text Add to dashboard Cite

Text-based Person Search (TBPS) aims to retrieve the person images using natural language descriptions. Recently, Contrastive Language Image Pretraining (CLIP), a universal large cross-modal vision-language pre-training model, has remarkably performed over various cross-modal downstream tasks due to its powerful cross-modal semantic learning capacity. TPBS, as a fine-grained cross-modal retrieval task, is also facing the rise of research on the CLIP-based TBPS. In order to explore the potential of the visual-language pre-training model for downstream TBPS tasks, this paper makes the first attempt to conduct a comprehensive empirical study of CLIP for TBPS and thus contribute a straightforward, incremental, yet strong TBPS-CLIP baseline to the TBPS community. We revisit critical design considerations under CLIP, including data augmentation and loss function. The model, with the aforementioned designs and practical training tricks, can attain satisfactory performance without any sophisticated modules. Also, we conduct the probing experiments of TBPS-CLIP in model generalization and model compression, demonstrating the effectiveness of TBPS-CLIP from various aspects. This work is expected to provide empirical insights and highlight future CLIP-based TBPS research.

show abstract

Section: Ablations Of Data Augmentationmentioning

confidence: 79%

An Empirical Study of CLIP for Text-Based Person Search

Cao,

Bai,

Zeng

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…We compare our proposed method CMAP with recent stateof-the-art methods, including: (1) Traditional pre-training methods that improve the accuracy of cross-modal matches by attention mechanisms and additional informative cues, such as LGUR (Shao et al 2022), LBUL (Wang et al 2022b); Several methods DSSL (Zhu et al 2021), AXM-Net (Farooq et al 2022), ISANet (Yan et al 2022b), CAIBC (Wang et al 2022a), RKT (Wu et al 2023) and SRCF (Suo et al 2022) propose some simple strategies to achieve distinct semantics for proper alignments.…”

Section: Overall Comparsion Resultsmentioning

confidence: 99%

Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval

Li,

He,

et al. 2024

AAAI

View full text Add to dashboard Cite

Text-based person retrieval aims at retrieving a specific pedestrian image from a gallery based on textual descriptions. The primary challenge is how to overcome the inherent heterogeneous modality gap in the situation of significant intra-class variation and minimal inter-class variation. Existing approaches commonly employ vision-language pre-training or attention mechanisms to learn appropriate cross-modal alignments from noise inputs. Despite commendable progress, current methods inevitably suffer from two defects: 1) Matching ambiguity, which mainly derives from unreliable matching pairs; 2) One-sided cross-modal alignments, stemming from the absence of exploring one-to-many correspondence, i.e., coarse-grained semantic alignment. These critical issues significantly deteriorate retrieval performance. To this end, we propose a novel framework termed Adaptive Uncertainty-based Learning (AUL) for text-based person retrieval from the uncertainty perspective. Specifically, our AUL framework consists of three key components: 1) Uncertainty-aware Matching Filtration that leverages Subjective Logic to effectively mitigate the disturbance of unreliable matching pairs and select high-confidence cross-modal matches for training; 2) Uncertainty-based Alignment Refinement, which not only simulates coarse-grained alignments by constructing uncertainty representations but also performs progressive learning to incorporate coarse- and fine-grained alignments properly; 3) Cross-modal Masked Modeling that aims at exploring more comprehensive relations between vision and language. Extensive experiments demonstrate that our AUL method consistently achieves state-of-the-art performance on three benchmark datasets in supervised, weakly supervised, and domain generalization settings. Our code is available at https://github.com/CFM-MSG/Code-AUL.

show abstract

“…The participants were expected to confirm if the images matched their respective labels. The identification model demanded of participants is to identification beyond color, as used in Wang et al (2022).…”

Section: Methodsmentioning

confidence: 99%

Arbitrary color usage in interface design and impact on intuitive use

Noble¹,

Matthias²

2023

J. Eng. Comput. Innov.

View full text Add to dashboard Cite

As the usage of computer systems is on the rise and role of user interfaces in usage space cannot be undermined, there has been tremendous research into interaction design schemes and interface design methodologies. However, research that directly addresses visual design approaches, which is the interplay between User Interface (UI) elements and the basic principles of design, is scarce. This particular investigation engaged participants (N=24) in an interactions scheme where color and labels of common fruit and vegetables are randomly altered. The aim is to ascertain if the arbitrary color attribution within the interaction scheme affects accuracy of outcome and/or interaction speed. Subjecting outcome various statistical analysis reveals interesting principle that broadly recapitulate all design decision in organizing elements. This study is an excerpt from a bigger instructional design process to develop a minimalist conceptual framework to support user interface design and evaluation.

show abstract

CAIBC: Capturing All-round Information Beyond Color for Text-based Person Retrieval

Cited by 52 publications

References 35 publications

An Empirical Study of CLIP for Text-Based Person Search

An Empirical Study of CLIP for Text-Based Person Search

Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval

Arbitrary color usage in interface design and impact on intuitive use

Contact Info

Product

Resources

About