A Survey on Contrastive Self-supervised Learning

Jaiswal, Ashish; Babu, Ashwin Ramesh; Zadeh, Mohammad Zaki; Banerjee, Debapriya; Makedon, Fillia

doi:10.48550/arxiv.2011.00362

Cited by 17 publications

(18 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has recently been reported that the performance of a model can be improved by training multiple tasks in the field of self-supervised learning [27]. As the task completion time is an additional task that can be provided for almost any task and it can be treated as a subtask in self-supervised learning, self-supervised learning can be achieved in almost any task using the proposed framework.…”

Section: B Examination Of Task Completion Time Repeatability For Comp...mentioning

confidence: 99%

Imitation learning for variable speed motion generation over multiple actions

Saigusa

Sasagawa

Sakaino

et al. 2021

IECON 2021 – 47th Annual Conference of the IEEE Industrial Electronics Society

View full text Add to dashboard Cite

Robot motion generation methods using machine learning have been studied in recent years. Bilateral controlbased imitation learning can imitate human motions using force information. By means of this method, variable speed motion generation that considers physical phenomena such as the inertial force and friction can be achieved. Previous research demonstrated that the complex relationship between the force and speed can be learned by using a neural network model. However, the previous study only focused on a simple reciprocating motion. To learn the complex relationship between the force and speed more accurately, it is necessary to learn multiple actions using many joints. In this paper, we propose a variable speed motion generation method for multiple motions. We considered four types of neural network models for the motion generation and determined the best model for multiple motions at variable speeds. Subsequently, we used the best model to evaluate the reproducibility of the task completion time for the input completion time command. The results revealed that the proposed method could change the task completion time according to the specified completion time command in multiple motions.

show abstract

Section: B Examination Of Task Completion Time Repeatability For Comp...mentioning

confidence: 99%

Imitation learning for variable speed motion generation over multiple actions

Saigusa

Sasagawa

Sakaino

et al. 2021

IECON 2021 – 47th Annual Conference of the IEEE Industrial Electronics Society

View full text Add to dashboard Cite

show abstract

“…Extending from TL, self-supervised learning (SSL) [27,28] is a technique where the pre-training stage uses only unlabeled data. Specifically, "pretext tasks" like image rotation prediction [29] and jigsaw puzzle solving [30] are invented for the data to provide its own supervision.…”

Section: Introductionmentioning

confidence: 99%

“…Specifically, "pretext tasks" like image rotation prediction [29] and jigsaw puzzle solving [30] are invented for the data to provide its own supervision. In particular, contrastive SSL [28] (or contrastive learning) is an increasingly popular technique where the pretext task is constructed as contrasting between two variations of a sample and other samples, where variations are derived using image transformations. The goal is for the pre-trained model to output embeddings where similar (differing) instances are closer (further) in the embedding metric space; this has proven to be highly effective for CV downstream tasks like classification, segmentation and detection.…”

Section: Introductionmentioning

confidence: 99%

Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science

Loh¹,

Christensen²,

Dangovski³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep learning techniques have been increasingly applied to the natural sciences, e.g., for property prediction and optimization or material discovery. A fundamental ingredient of such approaches is the vast quantity of labelled data needed to train the model; this poses severe challenges in data-scarce settings where obtaining labels requires substantial computational or labor resources. Here, we introduce surrogate-and invariance-boosted contrastive learning (SIB-CL), a deep learning framework which incorporates three "inexpensive" and easily obtainable auxiliary information sources to overcome data scarcity. Specifically, these are: 1) abundant unlabeled data, 2) prior knowledge of symmetries or invariances and 3) surrogate data obtained at near-zero cost. We demonstrate SIB-CL's effectiveness and generality on various scientific problems, e.g., predicting the density-ofstates of 2D photonic crystals and solving the 3D time-independent Schrödinger equation. SIB-CL consistently results in orders of magnitude reduction in the number of labels needed to achieve the same network accuracies.

show abstract

“…Recently, selfsupervised methods especially contrastive learning plays a vital role in the progress of deep learning due to data supervised itself. Besides, it can also alleviate the generalization error, spurious correlations and adversarial attacks [Jaiswal et al, 2021]. This kind of technology is applied in many fields.…”

Section: Introductionmentioning

confidence: 99%

Contrastive Pre-training for Imbalanced Corporate Credit Ratings

Feng¹,

Xue²

2021

Preprint

View full text Add to dashboard Cite

Corporate credit rating reflects the level of corporate credit and plays a crucial role in modern financial risk control. But real-world credit rating data usually shows long-tail distributions, which means heavy class imbalanced problem challenging the corporate credit rating system greatly.To tackle that, inspried by the recent advances of pre-train techniques in selfsupervised representation learning, we propose a novel framework named Contrastive Pre-training for Corporate Credit Rating (CP4CCR), which utilizes the self-surpervision for getting over class imbalance. Specifically, we propose to, in the first phase, exert constrastive self-superivised pretraining without label information, which want to learn a better class-agnostic initialization. During this phase, two self-supervised task are developed within CP4CCR: (i) Feature Masking (FM) and (ii) Feature Swapping(FS). In the second phase, we can train any standard corporate redit rating model initialized by the pre-trained network. Extensive experiments conducted on the Chinese public-listed corporate rating dataset, prove that CP4CCR can improve the performance of standard corporate credit rating models, especially for class with few samples.

show abstract

A Survey on Contrastive Self-supervised Learning

Cited by 17 publications

References 25 publications

Imitation learning for variable speed motion generation over multiple actions

Imitation learning for variable speed motion generation over multiple actions

Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science

Contrastive Pre-training for Imbalanced Corporate Credit Ratings

Contact Info

Product

Resources

About