Red Dragon AI at TextGraphs 2020 Shared Task : LIT : LSTM-Interleaved Transformer for Multi-Hop Explanation Ranking

Chia, Yew Ken; Witteveen, Sam; Andrews, Martin

doi:10.18653/v1/2020.textgraphs-1.14

Cited by 3 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Model compression is one approach to mitigating this issue. Various methods for compressing large-scale language models have been proposed in the last two years [8][9][10][11][12][13].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning

2022

View full text Add to dashboard Cite

Large-scale pre-trained language models such as BERT have brought much better performance to text classification. However, their large sizes can lead to sometimes prohibitively slow fine-tuning and inference. To alleviate this, various compression methods have been proposed; however, most of these methods solely consider reducing inference time, often ignoring significant increases in training time, and thus are even more resource consuming. In this article, we focus on lottery ticket extraction for the BERT architecture. Inspired by observations that representations at lower layers are often more useful for text classification, we propose that we can identify the winning ticket of BERT for binary text classification through adaptive truncation, i.e., a process that drops the top-k layers of the pre-trained model based on simple, fast computations. In this way, the cost for compressing and fine-tuning, as well as inference, can be vastly reduced. We present experiments on eight mainstream binary text classification datasets covering different input styles (i.e., single-text and text-pair), as well as different typical tasks (e.g., sentiment analysis, acceptability judgement, textual entailment, semantic similarity analysis and natural language inference). Compared with some strong baselines, our method saved 78.1% time and 31.7% memory on average, and up to 86.7 and 48% in extreme cases, respectively. We also saw good performance, often outperforming the original language model.

show abstract

“…Model compression is one approach to mitigating this issue. Various methods for compressing large-scale language models have been proposed in the last two years [8][9][10][11][12][13].…”

Section: Introductionmentioning

confidence: 99%

“…Model compression is one approach to mitigating this issue. Various methods for compressing large-scale language models have been proposed in the last two years [8][9][10][11][12][13]. From the perspective of downstream tasks, current model compression methods could be classified as task-agnostic and task-specific compression.…”

Section: Introductionmentioning

confidence: 99%

Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning

2022

View full text Add to dashboard Cite

show abstract

TextGraphs 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration

Jansen

Ustalov

2020

Proceedings of the Graph-Based Methods for Natural Language Processing (TextGraphs)

View full text Add to dashboard Cite

The 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration tasks participants with regenerating large detailed multi-fact explanations for standardized science exam questions. Given a question, correct answer, and knowledge base, models must rank each fact in the knowledge base such that facts most likely to appear in the explanation are ranked highest. Explanations consist of an average of 6 (and as many as 16) facts that span both core scientific knowledge and world knowledge, and form an explicit lexically-connected "explanation graph" describing how the facts interrelate. In this second iteration of the explanation regeneration shared task, participants are supplied with more than double the training and evaluation data of the first shared task, as well as a knowledge base nearly double in size, both of which expand into more challenging scientific topics that increase the difficulty of the task. In total 10 teams participated, and 5 teams submitted system description papers. The best-performing teams significantly increased state-of-the-art performance both in terms of ranking (mean average precision) and inference speed on this challenge task.

show abstract

TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration

Jansen

Thayaparan

Valentino

et al. 2021

Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15)

View full text Add to dashboard Cite

The Shared Task on Multi-Hop Inference for Explanation Regeneration asks participants to compose large multi-hop explanations to questions by assembling large chains of facts from a supporting knowledge base. While previous editions of this shared task aimed to evaluate explanatory completeness -finding a set of facts that form a complete inference chain, without gaps, to arrive from question to correct answer, this 2021 instantiation concentrates on the subtask of determining relevance in large multi-hop explanations. To this end, this edition of the shared task makes use of a large set of approximately 250k manual explanatory relevancy ratings that augment the 2020 shared task data. In this summary paper, we describe the details of the explanation regeneration task, the evaluation data, and the participating systems. Additionally, we perform a detailed analysis of participating systems, evaluating various aspects involved in the multi-hop inference process. The best performing system achieved an NDCG of 0.82 on this challenging task, substantially increasing performance over baseline methods by 32%, while also leaving significant room for future improvement.

show abstract

Red Dragon AI at TextGraphs 2020 Shared Task : LIT : LSTM-Interleaved Transformer for Multi-Hop Explanation Ranking

Cited by 3 publications

References 15 publications

Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning

Compressing BERT for Binary Text Classification via Adaptive Truncation before Fine-Tuning

TextGraphs 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration

TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration

Contact Info

Product

Resources

About