XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation

Mukherjee, Subhabrata; Awadallah, Ahmed Hassan; Gao, Jianfeng

doi:10.48550/arxiv.2106.04563

Cited by 6 publications

(10 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note, however, that the majority of the performance decrease due to the increasing depth is caused by only a single task -CoLA. This behaviour has previously been observed in the literature and is in line with other work trying to compress BERT behaviour into smaller models (Sun et al, 2019;Turc et al, 2019;Mukherjee et al, 2021). If we disregard CoLA, at least 98.6% of the predictive performance is preserved by all UltraFastBERT model.…”

Section: Resultssupporting

confidence: 90%

Periodic Extrapolative Generalisation in Neural Networks

Belcak¹,

Wattenhofer²

2022

2022 IEEE Symposium Series on Computational Intelligence (SSCI)

View full text Add to dashboard Cite

We break the linear link between the layer size and its inference cost by introducing the fast feedforward (FFF) architecture, a log-time alternative to feedforward networks. We show that FFFs give comparable performance to feedforward networks at an exponential fraction of their inference cost, are quicker to deliver performance compared to mixtureof-expert networks, and can readily take the place of either in transformers. Pushing FFFs to the absolute limit, we train a vision transformer to perform single-neuron inferences at the cost of only 5.8% performance decrease against the full-width variant.

show abstract

Section: Resultssupporting

confidence: 90%

Periodic Extrapolative Generalisation in Neural Networks

Belcak¹,

Wattenhofer²

2022

2022 IEEE Symposium Series on Computational Intelligence (SSCI)

View full text Add to dashboard Cite

show abstract

“…Regular expressions were still used in several cases to extract the key features. Then, di↵erent techniques were adopted to carry out the blocking itself, including sorted neighborhood, similarity joins [7,8], sentence encoding using BERT [10], a neural architecture based on a distilled transformer [15], and exploiting additional training data [16] to perform supervised contrastive learning [14]. These techniques were often followed by a pair/block cleaning and ranking step (based on intra-pair similarity) to comply with the submission structure.…”

Section: Solution Highlightsmentioning

confidence: 99%

Experiences and Lessons Learned from the SIGMOD Entity Resolution Programming Contests

et al. 2023

View full text Add to dashboard Cite

We report our experience in running three editions (2020, 2021, 2022) of the SIGMOD programming contest, a well-known event for students to engage in solving exciting data management problems. During this period we had the opportunity of introducing participants to the entity resolution task, which is of paramount importance in the data integration community. We aim at sharing the executive decisions, made by the people coauthoring this report, and the lessons learned.

show abstract

“…XDL has multiple variants available on 'huggingface' regarding encoder layers, hidden size, and attention heads. In this study, Xtremedistil-l6-h256-uncased has been used keeping the small model parameter in mind, as it only has 12.7 million tunable parameters [52].…”

Section: ) Xtremedistilmentioning

confidence: 99%

Depression Classification From Tweets Using Small Deep Transfer Learning Language Models

et al. 2022

View full text Add to dashboard Cite

Depression detection from social media texts such as Tweets or Facebook comments could be very beneficial as early detection of depression may even avoid extreme consequences of long-term depression i.e. suicide. In this study, depression intensity classification is performed using a labeled Twitter dataset. Further, this study makes a detailed performance evaluation of four transformer-based pre-trained small language models, particularly those having less than 15 million tunable parameters i.e. Electra Small Generator (ESG), Electra Small Discriminator (ESD), XtremeDistil-L6 (XDL) and Albert Base V2 (ABV) for classification of depression intensity using Tweets. The models are fine-tuned to get the best performance by applying different hyperparameters. The models are tested by classification of depression intensity of labeled tweets for three label classes i.e. 'severe', 'moderate', and 'mild' by downstream fine-tuning the parameters. Evaluation metrics such as accuracy, F1, precision, recall, and specificity are calculated to evaluate the performance of the models. Comparative analysis of these models is also done with a moderately larger model i.e. DistilBert which has 67 million tunable parameters for the same task with the same experimental settings. Results indicate that ESG outperforms all other models including DistilBert due to its better deep contextualized text representation as it gets the best F1 score of 89% with comparatively less training time. This study helps to achieve better classification performance of depression detection as well as to choose the best language model in terms of performance and less training time for Twitter-related downstream NLP tasks.INDEX TERMS Depression classification; transfer learning; transformer language models; public health

show abstract

XtremeDistilTransformers: Task Transfer for Task-agnostic Distillation

Cited by 6 publications

References 28 publications

Periodic Extrapolative Generalisation in Neural Networks

Periodic Extrapolative Generalisation in Neural Networks

Experiences and Lessons Learned from the SIGMOD Entity Resolution Programming Contests

Depression Classification From Tweets Using Small Deep Transfer Learning Language Models

Contact Info

Product

Resources

About