DimCL: Dimensional Contrastive Learning for Improving Self-Supervised Learning

Nguyen, Thanh Xuan; Pham, Trung X.; Zhang, Chaoning; Luu, Tung M.; Vu, Thang; Yoo, Chang D.

doi:10.1109/access.2023.3236087

Cited by 5 publications

(2 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…23 Contrastive learning is an important method in self-supervised learning that learns feature representations by comparing the similarity between samples. 24 Specifically, contrastive learning requires pairing different views or transformations of the same sample as positive pairs and views or transformations from different samples as negative pairs. By maximizing the similarity of positive pairs and minimizing the similarity of negative pairs, the model can learn feature representations with good discriminative ability.…”

Section: Self-supervised Learningmentioning

confidence: 99%

“…Contrastive learning is an important method in self-supervised learning that learns feature representations by comparing the similarity between samples 24 . Specifically, contrastive learning requires pairing different views or transformations of the same sample as positive pairs and views or transformations from different samples as negative pairs.…”

Section: Relation Workmentioning

confidence: 99%

See 1 more Smart Citation

Few-shot object detection based on self-supervised feature pyramid network

Lv,

Qi,

Shi

et al. 2024

J. Electron. Imag.

View full text Add to dashboard Cite

In few-shot object detection, the limited amount of labeled data fails to adequately represent all possible scenarios and objects. This limitation leads to the model's inability to fully learn the features and attributes of novel classes. In some cases, novel classes may be confused with base classes due to feature similarity, resulting in inaccurate detection results. During the two-stage fine-tuning phase, when there is a significant difference between the novel class and the base class data, the candidate boxes generated by the backbone network trained on the base class may not be suitable for the novel class targets. Therefore, it is a forward-looking research problem to explore how to mine novel class knowledge during the feature extraction process to supplement the disadvantage of having limited feature samples and improve the sensitivity in recognizing novel classes. To enrich the feature representation of novel classes, we propose a self-supervised feature pyramid network. This approach explores novel class attributes in the lower-level network, thereby encouraging the feature extractor to generate candidate boxes that are consistent with the novel class targets. The goal is to enhance the sensitivity of the backbone network in recognizing novel classes. We validate the effectiveness of our proposed framework by comparing it with state-of-the-art methods on two popular datasets and achieve an improvement of up to þ5.2% on the standard PASCAL VOC benchmark and a 1.4% boost on the challenging COCO benchmark.

show abstract

Section: Self-supervised Learningmentioning

confidence: 99%

Section: Relation Workmentioning

confidence: 99%

Few-shot object detection based on self-supervised feature pyramid network

Lv,

Qi,

Shi

et al. 2024

J. Electron. Imag.

View full text Add to dashboard Cite

show abstract

ABC-Learning: Attention-Boosted Contrastive Learning for unsupervised person re-identification

Zhang,

Wang,

Peng

2024

Engineering Applications of Artificial Intelligence

View full text Add to dashboard Cite

Self-Supervised Visual Representation Learning via Residual Momentum

Pham,

Niu,

Zhang

et al. 2023

IEEE Access

Self Cite

View full text Add to dashboard Cite

Self-supervised learning (SSL) has emerged as a promising approach for learning representations from unlabeled data. Momentum-based contrastive frameworks such as MoCo-v3 have shown remarkable success among the many SSL methods proposed in recent years. However, a significant gap in encoder representation exists between the online encoder (student) and the momentum encoder (teacher) in these frameworks, limiting the performance on downstream tasks. We identify this gap as a bottleneck often overlooked in existing frameworks and propose "residual momentum" that explicitly reduces the gap during training to encourage the student to learn representations closer to the teacher's. We also reveal that a similar technique, knowledge distillation (KD), to reduce the distribution gap with cross-entropybased loss in supervised learning is useless in the SSL context and demonstrate that the intra-representation gap measured by cosine similarity is crucial for EMA-based SSLs. Extensive experiments on different benchmark datasets and architectures demonstrate the superiority of our method compared to state-of-theart contrastive learning baselines. Specifically, our method outperforms MoCo-v3 0.7% top-1 in ImageNet, 2.82% on CIFAR-100, 1.8% AP, and 3.0% AP75 on VOC detection pre-trained on the COCO dataset; it also improves DenseCL with 0.5% AP (800ep) and 0.6% AP75 (1600ep). Our work highlights the importance of reducing the teacher-student intra-gap in momentum-based contrastive learning frameworks and provides a practical solution for improving the quality of learned representations.

show abstract

DimCL: Dimensional Contrastive Learning for Improving Self-Supervised Learning

Cited by 5 publications

References 56 publications

Few-shot object detection based on self-supervised feature pyramid network

Few-shot object detection based on self-supervised feature pyramid network

ABC-Learning: Attention-Boosted Contrastive Learning for unsupervised person re-identification

Self-Supervised Visual Representation Learning via Residual Momentum

Contact Info

Product

Resources

About