Aiming beyond the Obvious: Identifying Non-Obvious Cases in Semantic Similarity Datasets

Peinelt, Nicole; Liakata, Maria; Nguyen, Dong

doi:10.18653/v1/p19-1268

Cited by 11 publications

(10 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As we show, randomly combining sentences is insufficient. Sampling appropriate pairs has a decisive impact on performance which corresponds to recent findings on similar datasets (Peinelt et al, 2019).…”

Section: Related Worksupporting

confidence: 78%

Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

Nandan¹,

Reimers²,

Daxenberger³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

109

View full text Add to dashboard Cite

There are two approaches for pairwise sentence scoring: Cross-encoders, which perform full-attention over the input pair, and Bi-encoders, which map each input independently to a dense vector space. While crossencoders often achieve higher performance, they are too slow for many practical use cases. Bi-encoders, on the other hand, require substantial training data and fine-tuning over the target task to achieve competitive performance. We present a simple yet efficient data augmentation strategy called Augmented SBERT, where we use the cross-encoder to label a larger set of input pairs to augment the training data for the bi-encoder. We show that, in this process, selecting the sentence pairs is non-trivial and crucial for the success of the method. We evaluate our approach on multiple tasks (in-domain) as well as on a domain adaptation task. Augmented SBERT achieves an improvement of up to 6 points for in-domain and of up to 37 points for domain adaptation tasks compared to the original bi-encoder performance. 1

show abstract

Section: Related Worksupporting

confidence: 78%

Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

Nandan¹,

Reimers²,

Daxenberger³

et al. 2021

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

109

View full text Add to dashboard Cite

show abstract

“…SemEval C) than accuracy. We further report performance on difficult cases with non-obvious F1 score (Peinelt et al, 2019) which identifies challenging instances in the dataset based on lexical overlap and gold labels. Dodge et al (2020) recently showed that early stopping and random seeds can have considerable impact on the performance of finetuned BERT models.…”

Section: Tbert 31 Architecturementioning

confidence: 99%

tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection

Peinelt¹,

Nguyen²,

Liakata³

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

Self Cite

119

View full text Add to dashboard Cite

Semantic similarity detection is a fundamental task in natural language understanding. Adding topic information has been useful for previous feature-engineered semantic similarity models as well as neural models for other tasks. There is currently no standard way of combining topics with pretrained contextual representations such as BERT. We propose a novel topic-informed BERT-based architecture for pairwise semantic similarity detection and show that our model improves performance over strong neural baselines across a variety of English language datasets. We find that the addition of topics to BERT helps particularly with resolving domain-specific cases.

show abstract

“…Our main evaluation metric is the F1 score as this is more meaningful than accuracy for datasets with imbalanced label distributions (such as SemEval C, see Appendix A). We also report performance on difficult cases using the non-obvious F1 score (Peinelt et al, 2019). This metric distinguishes obvious from non-obvious instances in a dataset based on lexical overlap and gold labels, and calculates a separate F1 score for challenging cases.…”

Section: Metricsmentioning

confidence: 99%

GiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method

Peinelt¹,

Rei²,

Liakata³

2021

Findings of the Association for Computational Linguistics: EMNLP 2021

Self Cite

View full text Add to dashboard Cite

Large pre-trained language models such as BERT have been the driving force behind recent improvements across many NLP tasks. However, BERT is only trained to predict missing words -either through masking or next sentence prediction -and has no knowledge of lexical, syntactic or semantic information beyond what it picks up through unsupervised pre-training. We propose a novel method to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. When injecting counter-fitted and dependency-based embeddings, the performance improvements on multiple semantic similarity datasets indicate that such information is beneficial and currently missing from the original model. Our qualitative analysis shows that counter-fitted embedding injection is particularly beneficial, with notable improvements on examples that require synonym resolution.

show abstract

Aiming beyond the Obvious: Identifying Non-Obvious Cases in Semantic Similarity Datasets

Cited by 11 publications

References 14 publications

Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks

tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection

GiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method

Contact Info

Product

Resources

About