Hard Negative Sampling Strategies for Contrastive Representation Learning

Tabassum, Afrina; Wahed, Muntasir; Lourentzou, Ismini

doi:10.48550/arxiv.2206.01197

Cited by 5 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…49 Expanding on that idea, to enhance the quality of negative samples, we leverage the output predictions of the first-stage model and focus on sampling those hard negative labels which have slightly high probabilities of being positive, and we refer to this method as hard negative sampling. 50,51 The concept of hard negative sampling in this work is illustrated in Figure 4. Since only the reagents and solvents with high enough probabilities (larger than 0.3) would be ranked in the second-stage model, it should be put more emphasis on those hard negative chemicals.…”

Section: Methodsmentioning

confidence: 99%

Recommending Multiple Reaction Conditions Using Two-Stage Deep Neural Networks

Chen

2023

Preprint

View full text Add to dashboard Cite

The temperatures, prices of reagent and solvent and the yields influence the feasibility of the synthetic pathway. Therefore, predicting reaction condition is an important topic when designing a profitable synthetic pathway. For a single-step reaction, there are sometimes more than one suitable reaction contexts. Different reaction conditions result in different reaction rate and selectivity, and the design should comply with the requirement of the chemists. Providing diverse alternatives could help design the more economic synthesis pathway. However, recent literature has only tried to predict one best reaction condition. To improve this situation, we construct a twostage listwise ranking model to recommend multiple reaction conditions, and the ranking metrics are based on the yield level. The model is trained on the dataset consisting of ten representative types of reaction exported from Reaxys, and it recommends the reaction conditions with the top- 20 mean average precision (MAP) equal to 0.2723. The MAE of the temperature prediction is 11.1 °C. Besides, we used t-SNE to reduce the dimensionality of the embeddings and found that the model implicitly learns the pattern of reaction classification when predicting the reaction conditions.

show abstract

Section: Methodsmentioning

confidence: 99%

Recommending Multiple Reaction Conditions Using Two-Stage Deep Neural Networks

Chen

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…As the training continues, most samples become too easy, which contribute less to the training. Therefore, methods in (Tabassum et al, 2022;Robinson et al, 2020;Kalantidis et al, 2020; are proposed to use hard mining strategies to focus on informative samples. In this paper, considering the massive number of instances in M Ins , contrasting with all these instances naturally leads to redundancy and hinders the training.…”

Section: Graph Contrastive Learningmentioning

confidence: 99%

Graph Contrastive Learning for Skeleton-based Action Recognition

Huang¹,

Zhang²,

Bao³

et al. 2023

Preprint

View full text Add to dashboard Cite

In the field of skeleton-based action recognition, current top-performing graph convolutional networks (GCNs) exploit intra-sequence context to construct adaptive graphs for feature aggregation. However, we argue that such context is still local since the rich cross-sequence relations have not been explicitly investigated. In this paper, we propose a graph contrastive learning framework for skeletonbased action recognition (SkeletonGCL) to explore the global context across all sequences. In specific, SkeletonGCL associates graph learning across sequences by enforcing graphs to be class-discriminative, i.e., intra-class compact and interclass dispersed, which improves the GCN capacity to distinguish various action patterns. Besides, two memory banks are designed to enrich cross-sequence context from two complementary levels, i.e., instance and semantic levels, enabling graph contrastive learning in multiple context scales. Consequently, SkeletonGCL establishes a new training paradigm, and it can be seamlessly incorporated into current GCNs. Without loss of generality, we combine SkeletonGCL with three GCNs (2S-ACGN, CTR-GCN, and InfoGCN), and achieve consistent improvements on NTU60, NTU120, and NW-UCLA benchmarks. The source code will be available at https://github.com/OliverHxh/SkeletonGCL.

show abstract

“…Additionally, HCL [39] revises the original InfoNCE objective by assigning higher weights for hard negatives among the mini-batch. Recently, Un-ReMix [44] is proposed to sample hard negatives by effectively capturing aspects of anchor similarity, representativeness, and model uncertainty. However, such locally sampled hard negatives cannot exploit hard negatives sufficiently from the dataset.…”

Section: Related Workmentioning

confidence: 99%

BatchSampler: Sampling Mini-Batches for Contrastive Learning in Vision, Language, and Graphs

Yang

Ming

Dong

et al. 2023

Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

In-Batch contrastive learning is a state-of-the-art self-supervised method that brings semantically-similar instances close while pushing dissimilar instances apart within a mini-batch. Its key to success is the negative sharing strategy, in which every instance serves as a negative for the others within the mini-batch. Recent studies aim to improve performance by sampling hard negatives within the current mini-batch, whose quality is bounded by the mini-batch itself. In this work, we propose to improve contrastive learning by sampling mini-batches from the input data. We present BatchSampler 1 to sample mini-batches of hard-to-distinguish (i.e., hard and true negatives to each other) instances. To make each mini-batch have fewer false negatives, we design the proximity graph of randomly-selected instances. To form the mini-batch, we leverage random walk with restart on the proximity graph to help sample hard-to-distinguish instances. BatchSampler is a simple and general technique that can be directly plugged into existing contrastive learning models in vision, language, and graphs. Extensive experiments on datasets of three modalities show that BatchSampler can consistently improve the performance of powerful contrastive models, as shown by significant improvements of SimCLR on ImageNet-100, SimCSE on STS (language), and GraphCL and MVGRL on graph datasets.

show abstract

Hard Negative Sampling Strategies for Contrastive Representation Learning

Cited by 5 publications

References 0 publications

Recommending Multiple Reaction Conditions Using Two-Stage Deep Neural Networks

Recommending Multiple Reaction Conditions Using Two-Stage Deep Neural Networks

Graph Contrastive Learning for Skeleton-based Action Recognition

BatchSampler: Sampling Mini-Batches for Contrastive Learning in Vision, Language, and Graphs

Contact Info

Product

Resources

About