Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level

Zhong, Ruiqi; Ghosh, Dhruba; Klein, Dan; Steinhardt, Jacob

doi:10.48550/arxiv.2105.06020

Cited by 4 publications

(6 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1. 10 BERT base models pretrained with different random seeds but not finetuned for particular tasks, released by Zhong et al [37] 4 . 2.…”

Section: Models We Studymentioning

confidence: 99%

“…2. 10 BERT medium models that were initialized from pretrained models released by Zhong et al [37], that we further finetuned on MNLI with 10 different finetuning seeds (100 models total). 3.…”

Section: Models We Studymentioning

confidence: 99%

“…We fine-tuned models from Zhong et al [37] and the original BERT models from Devlin et al [8] on three tasks -Quora Question Pairs (QQP) 10 , Multi-Genre Natural Language Inference (MNLI; Williams et al [36]), and the Stanford Sentiment Treebank (SST-2; Socher et al [29]), and show each model's accuracy on these tasks in Table 2. Our models generally have comparable accuracy.…”

Section: Appendix a Bert Finetuning Detailsmentioning

confidence: 99%

See 2 more Smart Citations

Grounding Representation Similarity with Statistical Testing

Ding¹,

Denain²,

Steinhardt

2021

Preprint

Self Cite

View full text Add to dashboard Cite

To understand neural network behavior, recent works quantitatively compare different networks' learned representations using canonical correlation analysis (CCA), centered kernel alignment (CKA), and other dissimilarity measures. Unfortunately, these widely used measures often disagree on fundamental observations, such as whether deep networks differing only in random initialization learn similar representations. These disagreements raise the question: which, if any, of these dissimilarity measures should we believe? We provide a framework to ground this question through a concrete test: measures should have sensitivity to changes that affect functional behavior, and specificity against changes that do not. We quantify this through a variety of functional behaviors including probing accuracy and robustness to distribution shift, and examine changes such as varying random initialization and deleting principal components. We find that current metrics exhibit different weaknesses, note that a classical baseline performs surprisingly well, and highlight settings where all metrics appear to fail, thus providing a challenge set for further improvement.Preprint. Under review.

show abstract

“…1. 10 BERT base models pretrained with different random seeds but not finetuned for particular tasks, released by Zhong et al [37] 4 . 2.…”

Section: Models We Studymentioning

confidence: 99%

Section: Models We Studymentioning

confidence: 99%

Section: Appendix a Bert Finetuning Detailsmentioning

confidence: 99%

See 1 more Smart Citation

Grounding Representation Similarity with Statistical Testing

Ding¹,

Denain²,

Steinhardt

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Aggregation when instance-level information is available. As illustrated by Zhong et al (2021); Ruder (2021), a fine-grained understanding of the model performance should include instance-level scores. If taking the mean is quite natural in the classification setting, this is not always the case, as recently pointed out by (Peyrard et al, 2021) in the NLG setting.…”

Section: Work In Progressmentioning

confidence: 99%

What are the best systems? New perspectives on NLP Benchmarking

Colombo¹,

Noiry²,

Irurozki³

et al. 2022

Preprint

View full text Add to dashboard Cite

In Machine Learning, a benchmark refers to an ensemble of datasets associated with one or multiple metrics together with a way to aggregate different systems performances. They are instrumental in (i) assessing the progress of new methods along different axes and (ii) selecting the best systems for practical use. This is particularly the case for NLP with the development of large pre-trained models (e.g. GPT, BERT) that are expected to generalize well on a variety of tasks. While the community mainly focused on developing new datasets and metrics, there has been little interest in the aggregation procedure, which is often reduced to a simple average over various performance measures. However, this procedure can be problematic when the metrics are on a different scale, which may lead to spurious conclusions. This paper proposes a new procedure to rank systems based on their performance across different tasks. Motivated by the social choice theory, the final system ordering is obtained through aggregating the rankings induced by each task and is theoretically grounded. We conduct extensive numerical experiments (on over 270k scores) to assess the soundness of our approach both on synthetic and real scores (e.g. GLUE, EXTREM, SE-VAL, TAC, FLICKR). In particular, we show that our method yields different conclusions on stateof-the-art systems than the mean-aggregation procedure while being both more reliable and robust.

show abstract

“…Noticeably, BioBERT large (M5.1-3) performs worse than its base counterparts, especially in the Int class, which warrants further investigation as no interpretable patterns could be found. However, recent findings [34] suggest that fine-tuning noise increases with model size and that instance-level accuracy has momentum leading to larger models having higher variance due to the fine-tuning seed.…”

Section: The Importance Of the Pre-trained Language Model Domainmentioning

confidence: 99%

A Message Passing Approach to Biomedical Relation Classification for Drug–Drug Interactions

2022

View full text Add to dashboard Cite

The task of extracting drug entities and possible interactions between drug pairings is known as Drug–Drug Interaction (DDI) extraction. Computer-assisted DDI extraction with Machine Learning techniques can help streamline this expensive and time-consuming process during the drug development cycle. Over the years, a variety of both traditional and Neural Network-based techniques for the extraction of DDIs have been proposed. Despite the introduction of several successful strategies, obtaining high classification accuracy is still an area where further progress can be made. In this work, we present a novel Knowledge Graph (KG) based approach that utilizes a unique graph structure in combination with a Transformer-based Language Model and Graph Neural Networks to classify DDIs from biomedical literature. The KG is constructed to model the knowledge of the DDI Extraction 2013 benchmark dataset, without the inclusion of additional external information sources. Each drug pair is classified based on the context of the sentence it was found in, by utilizing transfer knowledge in the form of semantic representations from domain-adapted BioBERT weights that serve as the initial KG states. The proposed approach was evaluated on the DDI classification task of the same dataset and achieved a F1-score of 79.14% on the four positive classes, outperforming the current state-of-the-art approach.

show abstract

Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level

Cited by 4 publications

References 17 publications

Grounding Representation Similarity with Statistical Testing

Grounding Representation Similarity with Statistical Testing

What are the best systems? New perspectives on NLP Benchmarking

A Message Passing Approach to Biomedical Relation Classification for Drug–Drug Interactions

Contact Info

Product

Resources

About