Contrastive Representation Learning: A Framework and Review

Le-Khac, Phuc H.; Healy, Graham; Smeaton, Alan F.

doi:10.1109/access.2020.3031549

Cited by 507 publications

(220 citation statements)

References 90 publications

Supporting

Mentioning

178

Contrasting

Order By: Relevance

“…Contrastive learning has been successfully applied to unsupervised representation learning [19]- [21], [32], [33] in recent years. Typical contrastive losses are inspired by noise contrastive estimation [34] or N-pair losses [35], aiming to pull positive samples together while push away negative samples.…”

Section: Contrastive Learningmentioning

confidence: 99%

Graph-Induced Contrastive Learning for Intra-Camera Supervised Person Re-Identification

et al. 2021

View full text Add to dashboard Cite

Intra-camera supervision (ICS) for person re-identification (Re-ID) assumes that identity labels are independently annotated within each camera view and no inter-camera identity association is labeled. It is a new setting proposed recently to reduce the burden of annotation while expect to maintain desirable Re-ID performance. However, the lack of inter-camera labels makes the ICS Re-ID problem much more challenging than the fully supervised counterpart. By investigating the characteristics of ICS, this paper proposes a graph-induced contrastive learning (GCL) approach to address this issue. More specifically, we first formulate the cross-camera ID association task as a graph partitioning problem subjected to ICS-specific constraints and design a greedy agglomeration algorithm to solve it. Then, we propose a graph-induced contrastive loss that unifies both intra-and inter-camera learning into a contrastive learning framework to learn a Re-ID model. The cross-camera ID association step and the Re-ID model contrastive learning step are alternatively iterated, by which we progressively obtain a highly discriminative Re-ID model. Extensive experiments on three large-scale datasets show that our approach outperforms all previous ICS works. Especially, it gains 15.7% Rank-1 and 14.3% mAP improvements on the challenging MSMT17 dataset. Moreover, our approach performs even comparable to state-of-the-art fully supervised methods on all of the three datasets.

show abstract

Section: Contrastive Learningmentioning

confidence: 99%

Graph-Induced Contrastive Learning for Intra-Camera Supervised Person Re-Identification

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Nowadays, transformer architectures (e.g., [ 3 , 11 , 12 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 ]) are seen as the state of the art for deep learning of the type of present interest. As per the definition of the contrastive learning framework mentioned in [ 61 , 66 ], we add an extra autoencoder in which the encoder behaves as a projection head. The outputs of the transformer encoder, which we regard as representations, are to be of a higher dimension.…”

Section: Introductionmentioning

confidence: 99%

FragNet, a Contrastive Learning-Based Transformer Model for Clustering, Interpreting, Visualizing, and Navigating Chemical Space

Shrivastava

Kell

2021

Molecules

View full text Add to dashboard Cite

The question of molecular similarity is core in cheminformatics and is usually assessed via a pairwise comparison based on vectors of properties or molecular fingerprints. We recently exploited variational autoencoders to embed 6M molecules in a chemical space, such that their (Euclidean) distance within the latent space so formed could be assessed within the framework of the entire molecular set. However, the standard objective function used did not seek to manipulate the latent space so as to cluster the molecules based on any perceived similarity. Using a set of some 160,000 molecules of biological relevance, we here bring together three modern elements of deep learning to create a novel and disentangled latent space, viz transformers, contrastive learning, and an embedded autoencoder. The effective dimensionality of the latent space was varied such that clear separation of individual types of molecules could be observed within individual dimensions of the latent space. The capacity of the network was such that many dimensions were not populated at all. As before, we assessed the utility of the representation by comparing clozapine with its near neighbors, and we also did the same for various antibiotics related to flucloxacillin. Transformers, especially when as here coupled with contrastive learning, effectively provide one-shot learning and lead to a successful and disentangled representation of molecular latent spaces that at once uses the entire training set in their construction while allowing “similar” molecules to cluster together in an effective and interpretable way.

show abstract

“…In the natural language and image processing fields, both supervised and unsupervised approaches enabled the creation of powerful pre-trained models, that are often employed in many different tasks [8,9,10]. Contrastive learning recently got a lot of attention due to its success for the unsupervised pre-training of DNNs, enabling to learn flexible representation without the need of having labels associated with the content [10,11]. Only the definition of positive and negative data pairs is required in order to learn a model that will produce a latent space reflecting semantic characteristics.…”

Section: Introductionmentioning

confidence: 99%

Learning Contextual Tag Embeddings for Cross-Modal Alignment of Audio and Tags

Favory

Drossos

Virtanen

et al. 2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Self-supervised audio representation learning offers an attractive alternative for obtaining generic audio embeddings, capable to be employed into various downstream tasks. Published approaches that consider both audio and words/tags associated with audio do not employ text processing models that are capable to generalize to tags unknown during training. In this work we propose a method for learning audio representations using an audio autoencoder (AAE), a general word embeddings model (WEM), and a multi-head self-attention (MHA) mechanism. MHA attends on the output of the WEM, providing a contextualized representation of the tags associated with the audio, and we align the output of MHA with the output of the encoder of AAE using a contrastive loss. We jointly optimize AAE and MHA and we evaluate the audio representations (i.e. the output of the encoder of AAE) by utilizing them in three different downstream tasks, namely sound, music genre, and music instrument classification. Our results show that employing multi-head self-attention with multiple heads in the tag-based network can induce better learned audio representations.

show abstract

Contrastive Representation Learning: A Framework and Review

Cited by 507 publications

References 90 publications

Graph-Induced Contrastive Learning for Intra-Camera Supervised Person Re-Identification

Graph-Induced Contrastive Learning for Intra-Camera Supervised Person Re-Identification

FragNet, a Contrastive Learning-Based Transformer Model for Clustering, Interpreting, Visualizing, and Navigating Chemical Space

Learning Contextual Tag Embeddings for Cross-Modal Alignment of Audio and Tags

Contact Info

Product

Resources

About