A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models

Mokrii, Iurii; Boytsov, Leonid; Braslavski, Pavel

doi:10.1145/3404835.3463093

Cited by 21 publications

(20 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Table 2, we report the BEIR results for our two CoCondenser scenarios, reaching state-of-the-art results on zero-shot evaluation, strengthening the observation that sparse retrieval models seem to be better able to generalize [5,17,26]. We also report in Table 3 the results of the two previous approaches combined with BM25 (sum); additional gains can be obtained, showing that pure lexical approaches are still somehow complementary to sparse neural models, especially in a zero-shot setting.…”

Section: Modelsupporting

confidence: 62%

From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective

Formal¹,

Lassance²,

Piwowarski³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural retrievers based on dense representations combined with Approximate Nearest Neighbors search have recently received a lot of attention, owing their success to distillation and/or better sampling of examples for training -while still relying on the same backbone architecture. In the meantime, sparse representation learning fueled by traditional inverted indexing techniques has seen a growing interest, inheriting from desirable IR priors such as explicit lexical matching. While some architectural variants have been proposed, a lesser effort has been put in the training of such models. In this work, we build on SPLADE -a sparse expansion-based retriever -and show to which extent it is able to benefit from the same training improvements as dense models, by studying the effect of distillation, hard-negative mining as well as the Pre-trained Language Model initialization. We furthermore study the link between effectiveness and efficiency, on in-domain and zero-shot settings, leading to state-of-the-art results in both scenarios for sufficiently expressive models. CCS CONCEPTS• Information systems → Retrieval models and ranking.

show abstract

Section: Modelsupporting

confidence: 62%

From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective

Formal¹,

Lassance²,

Piwowarski³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Humor (Forough and Momtazi, 2021) , label smoothing (Müller et al, 2019), and pseudo labeling (Mokrii et al, 2021). However, we argue that SAM is the main module to attain the exceptional performance.…”

Section: Methodsmentioning

confidence: 67%

Sequential Attention Module for Natural Language Processing

Zhou¹,

Ma²,

Yang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, large pre-trained neural language models have attained remarkable performance on many downstream natural language processing (NLP) applications via fine-tuning. In this paper, we target at how to further improve the token representations on the language models. We, therefore, propose a simple yet effective plug-and-play module, Sequential Attention Module (SAM), on the token embeddings learned from a pre-trained language model. Our proposed SAM consists of two main attention modules deployed sequentially: Featurewise Attention Module (FAM) and Token-wise Attention Module (TAM). More specifically, FAM can effectively identify the importance of features at each dimension and promote the effect via dot-product on the original token embeddings for downstream NLP applications. Meanwhile, TAM can further re-weight the features at the token-wise level. Moreover, we propose an adaptive filter on FAM to prevent noise impact and increase information absorption. Finally, we conduct extensive experiments to demonstrate the advantages and properties of our proposed SAM. We first show how SAM plays a primary role in the champion solution of two subtasks of SemEval'21 Task 7. After that, we apply SAM on sentiment analysis and three popular NLP tasks and demonstrate that SAM consistently outperforms the state-of-the-art baselines.

show abstract

“…There is limited evidence regarding the use of machine learning to score free‐text medical professionalism questions 3 . With the development of modern machine learning techniques, such as Bidirectional Encoder Representations from Transformers (BERT), it is possible that machine learning may be able to assist with this task 8–10 …”

Section: Number Of Marks Model Professionalism Mean Classification Ac...mentioning

confidence: 99%

Professionalism and clinical short answer question marking with machine learning

Lam

Blacketer

et al. 2022

Internal Medicine Journal

View full text Add to dashboard Cite

Machine learning may assist in medical student evaluation. This study involved scoring short answer questions administered at three centres. Bidirectional encoder representations from transformers were particularly effective for professionalism question scoring (accuracy ranging from 41.6% to 92.5%). In the scoring of 3‐mark professionalism questions, as compared with clinical questions, machine learning had a lower classification accuracy (P < 0.05). The role of machine learning in medical professionalism evaluation warrants further investigation.

show abstract

A Systematic Evaluation of Transfer Learning and Pseudo-labeling with BERT-based Ranking Models

Cited by 21 publications

References 30 publications

From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective

From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective

Sequential Attention Module for Natural Language Processing

Professionalism and clinical short answer question marking with machine learning

Contact Info

Product

Resources

About