Universum-Inspired Supervised Contrastive Learning

Han, Aiyang; Chen, Songcan

doi:10.1007/978-3-031-25198-6_34

Cited by 6 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our double Universums are Mixup Induced Universums (Han and Chen 2023;Vapnik 2006;Chapelle et al 2007) in both instance-and temporal-wise, which is anchor-specific mixing in the embedding space that mixes the specific positive feature (anchor) with the negative features for unannotated datasets. Let i be the index of the input time series sample and t be the timestamp.…”

Section: Double Universum Learningmentioning

confidence: 99%

TimesURL: Self-Supervised Contrastive Learning for Universal Time Series Representation Learning

Liu,

Chen

2024

AAAI

View full text Add to dashboard Cite

Learning universal time series representations applicable to various types of downstream tasks is challenging but valuable in real applications. Recently, researchers have attempted to leverage the success of self-supervised contrastive learning (SSCL) in Computer Vision(CV) and Natural Language Processing(NLP) to tackle time series representation. Nevertheless, due to the special temporal characteristics, relying solely on empirical guidance from other domains may be ineffective for time series and difficult to adapt to multiple downstream tasks. To this end, we review three parts involved in SSCL including 1) designing augmentation methods for positive pairs, 2) constructing (hard) negative pairs, and 3) designing SSCL loss. For 1) and 2), we find that unsuitable positive and negative pair construction may introduce inappropriate inductive biases, which neither preserve temporal properties nor provide sufficient discriminative features. For 3), just exploring segment- or instance-level semantics information is not enough for learning universal representation. To remedy the above issues, we propose a novel self-supervised framework named TimesURL. Specifically, we first introduce a frequency-temporal-based augmentation to keep the temporal property unchanged. And then, we construct double Universums as a special kind of hard negative to guide better contrastive learning. Additionally, we introduce time reconstruction as a joint optimization objective with contrastive learning to capture both segment-level and instance-level information. As a result, TimesURL can learn high-quality universal representations and achieve state-of-the-art performance in 6 different downstream tasks, including short- and long-term forecasting, imputation, classification, anomaly detection and transfer learning.

show abstract

Section: Double Universum Learningmentioning

confidence: 99%

TimesURL: Self-Supervised Contrastive Learning for Universal Time Series Representation Learning

Liu,

Chen

2024

AAAI

View full text Add to dashboard Cite

show abstract

“…These two advantages of SCL settings provide more efficient feature learning over the SSCL approach. Several studies in the audio domain have effectively applied SCL, for instance, in environmental sound classification [31], voice activity detection [32], accented speech recognition [33], and musical onset detection [34], exhibiting superior performance when compared to models trained using cross-entropy.…”

Section: Related Workmentioning

confidence: 99%

Identification of Non-Speaking and Minimal-Speaking Individuals Using Nonverbal Vocalizations

Tran,

Tsai

2024

IEEE Access

View full text Add to dashboard Cite

Speech remains a prevalent mode of communication powering various intelligent functions in human-computer interaction applications, notably in Speaker/Person Identification (PID) systems. However, there is a considerable population of Non-speaking and Minimal-speaking (NMS) individuals, who heavily rely on nonverbal vocalizations for communication, and the existing speech-based PID systems may not be suitable for users from this community. This study delves into the use of nonverbal vocalizations to identify NMS subjects, termed as NMS-PID, and explores the feasibility of developing an identification system, namely S-NMS-PID, that accommodates both speaking users (with speech input) and NMS users (with nonverbal-vocalization input). Leveraging the recently published ReCANVo dataset of NMS nonverbal vocalizations and our speech dataset, our experiments with multiple networks and acoustic features demonstrate promising results for NMS-PID and S-NMS-PID, evident in average accuracies ranging from 70% to 92%. The proposed convolutional recurrent neural network-based model, despite its smaller size, achieves results nearly on par with much deeper models such as VGG16 and ResNet50. Our findings also highlight the efficacy of Mel-frequency cepstral coefficients features compared to the spectrogram features. Furthermore, a two-step training strategy involving supervised contrastive learning for representation learning followed by fine-tuning with cross-entropy loss significantly enhances robustness and accuracy, particularly in classifying data from minority classes, enhancing overall performance. This study's outcomes hold potential for tailoring human-computer interaction applications specifically for NMS users. Implementing NMS-PID and S-NMS-PID in security and authentication processes ensures secure and reliable user identification across diverse platforms, transcending sole reliance on speech-based methods.

show abstract

“…Our double Universums are Mixup Induced Universums (Han and Chen 2023;Vapnik 2006;Chapelle et al 2007) in both instance-and temporal-wise, which is anchor-specific Let i be the index of the input time series sample and t be the timestamp. r i,t and r ′ i,t denote the representations for the same timestamp t but from two augmentation of x i .…”

Section: Double Universum Learningmentioning

confidence: 99%

Dual-channel graph contrastive learning for self-supervised graph-level representation learning

Luo

Dong

Zheng

et al. 2023

Pattern Recognition

View full text Add to dashboard Cite

Universum-Inspired Supervised Contrastive Learning

Cited by 6 publications

References 16 publications

TimesURL: Self-Supervised Contrastive Learning for Universal Time Series Representation Learning

TimesURL: Self-Supervised Contrastive Learning for Universal Time Series Representation Learning

Identification of Non-Speaking and Minimal-Speaking Individuals Using Nonverbal Vocalizations

Dual-channel graph contrastive learning for self-supervised graph-level representation learning

Contact Info

Product

Resources

About