SPBERTQA: A Two-Stage Question Answering System Based on Sentence Transformers for Medical Texts

Nguyen, Nhung T. H.; Ha, Phuong Phan-Dieu; Nguyễn, Luân; Nguyen, Kiet Van; Nguyen, Ngan Luu-Thuy

doi:10.1007/978-3-031-10986-7_30

Cited by 8 publications

(2 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also provided the theoretical proof that contrastive learning induces models to weight each word by KL(w). These provide insights into why contrastive-based sentence encoders succeed in a wide range of tasks, such as information retrieval (Muennighoff, 2022) and question answering (Nguyen et al, 2022), where emphasizing some informative words is effective. Besides sentence encoders, investigating the word weighting of retrieval models is an interesting future direction.…”

Section: Discussionmentioning

confidence: 99%

“…Embedding a sentence into a point in a highdimensional continuous space plays a foundational role in the natural language processing (NLP) (Arora et al, 2017;Reimers and Gurevych, 2019;Chuang et al, 2022, etc.). Such sentence embedding methods can also embed text of various types and lengths, such as queries, passages, and paragraphs; therefore, they are widely used in diverse applications such as information retrieval (Karpukhin et al, 2020;Muennighoff, 2022), question answering (Nguyen et al, 2022), and retrieval-augmented generation (Chase, 2023).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Contrastive Learning-based Sentence Encoders Implicitly Weight Informative Words

Kurita,

Kobayashi,

Yokoi

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

The performance of sentence encoders can be significantly improved through the simple practice of fine-tuning using contrastive loss. A natural question arises: what characteristics do models acquire during contrastive learning? This paper theoretically and experimentally shows that contrastive-based sentence encoders implicitly weight words based on informationtheoretic quantities; that is, more informative words receive greater weight, while others receive less. The theory states that, in the lower bound of the optimal value of the contrastive learning objective, the norm of word embedding reflects the information gain associated with the distribution of surrounding words. We also conduct comprehensive experiments using various models, multiple datasets, two methods to measure the implicit weighting of models (Integrated Gradients and SHAP), and two information-theoretic quantities (information gain and self-information). The results provide empirical evidence that contrastive fine-tuning emphasizes informative words.

show abstract

Section: Discussionmentioning

confidence: 99%