ReasonBERT: Pre-trained to Reason with Distant Supervision

Deng, Xiang; Su, Yu Ru; Lees, Alyssa; Wu, You; Yu, Cong; Sun, Huan

doi:10.18653/v1/2021.emnlp-main.494

Cited by 11 publications

(12 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It consists of various sub-skills including commonsense reasoning (Zellers et al, 2018;Talmor et al, 2019;Bhagavatula et al, 2019), numerical reasoning (Dua et al, 2019), arithmetic reasoning (Koncel-Kedziorski et al, 2015;Roy and Roth, 2016;Miao et al, 2020;Cobbe et al, 2021), logical reasoning (Yu et al, 2020), tabular reasoning (Zhu et al, 2021), and so on. Previous efforts in machine learning exploited symbolic systems (Mihaylov and Frank, 2018;Ding et al, 2019;Wang et al, 2022b,a) and pre-training strategies (Deng et al, 2021;Asai and Hajishirzi, 2020;Pi et al, 2022). Recently, large language models with chain-of-thought prompting (Wei et al, 2022b;Wang et al, 2022c;Zhou et al, 2022;Zhang et al, 2022b) demonstrate promising reasoning abilities with appropriately designed prompts, achieving competitive performance on several benchmarks.…”

Section: Reasoning Abilitymentioning

confidence: 99%

Active Prompting with Chain-of-Thought for Large Language Models

Diao¹,

Wang²,

Ye³

et al. 2023

Preprint

View full text Add to dashboard Cite

The increasing scale of large language models (LLMs) brings emergent abilities to various complex tasks requiring reasoning, such as arithmetic and commonsense reasoning. It is known that the effective design of task-specific prompts is critical for LLMs' ability to produce high-quality answers. In particular, an effective approach for complex question-andanswer tasks is example-based prompting with chain-of-thought (CoT) reasoning, which significantly improves the performance of LLMs. However, current CoT methods rely on a fixed set of human-annotated exemplars, which are not necessarily the most effective examples for different tasks. This paper proposes a new method, Active-Prompt, to adapt LLMs to different tasks with task-specific example prompts (annotated with human-designed CoT reasoning). For this purpose, we propose a solution to the key problem of determining which questions are the most important and helpful ones to annotate from a pool of taskspecific queries. By borrowing ideas from the related problem of uncertainty-based active learning, we introduce several metrics to characterize the uncertainty so as to select the most uncertain questions for annotation. Experimental results demonstrate the superiority of our proposed method, achieving state-ofthe-art on eight complex reasoning tasks. Further analyses of different uncertainty metrics, pool sizes, zero-shot learning, and accuracyuncertainty relationship demonstrate the effectiveness of our method. 1

show abstract

Section: Reasoning Abilitymentioning

confidence: 99%

Active Prompting with Chain-of-Thought for Large Language Models

Diao¹,

Wang²,

Ye³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, 1. Smarter span selection: We only consider spans selected uniformly at random for generality, but mixing in semantically or syntactically meaningful spans [Donahue et al, 2020, Joshi et al, 2020, Deng et al, 2021 can considerably improve infilling performance. In Section 4.5, we see that training on line-level spans instead of character-level spans improves line-based infilling results.…”

Section: Future Directionsmentioning

confidence: 99%

Efficient Training of Language Models to Fill in the Middle

Bavarian¹,

Jun²,

Tezak³

et al. 2022

Preprint

View full text Add to dashboard Cite

We show that autoregressive language models can learn to infill text after we apply a straightforward transformation to the dataset, which simply moves a span of text from the middle of a document to its end. While this data augmentation has garnered much interest in recent years, we provide extensive evidence that training models with a large fraction of data transformed in this way does not harm the original left-to-right generative capability, as measured by perplexity and sampling evaluations across a wide range of scales. Given the usefulness, simplicity, and efficiency of training models to fill-in-the-middle (FIM), we suggest that future autoregressive language models be trained with FIM by default. To this end, we run a series of ablations on key hyperparameters, such as the data transformation frequency, the structure of the transformation, and the method of selecting the infill span. We use these ablations to prescribe strong default settings and best practices to train FIM models. We have released our best infilling model trained with best practices in our API, and release our infilling benchmarks to aid future research.

show abstract

“…(2) The latter is task-oriented, which includes several well designed pre-training or fine-tuning tasks (Jiao et al, 2021;Deng et al, 2021). However, none of these works target cross-lingual areas, especially for xSL tasks.…”

Section: Related Workmentioning

confidence: 99%

Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling

Shou¹,

Gong²,

Pei³

et al. 2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

Large-scale cross-lingual pre-trained language models (xPLMs) have shown effectiveness in cross-lingual sequence labeling tasks (xSL), such as cross-lingual machine reading comprehension (xMRC) by transferring knowledge from a high-resource language to low-resource languages. Despite the great success, we draw an empirical observation that there is a training objective gap between pre-training and finetuning stages: e.g., mask language modeling objective requires local understanding of the masked token and the span-extraction objective requires global understanding and reasoning of the input passage/paragraph and question, leading to the discrepancy between pretraining and xMRC. In this paper, we first design a pre-training task tailored for xSL named Cross-lingual Language Informative Span Masking (CLISM) to eliminate the objective gap in a self-supervised manner. Second, we present ContrAstive-Consistency Regularization (CACR), which utilizes contrastive learning to encourage the consistency between representations of input parallel sequences via unsupervised cross-lingual instance-wise training signals during pre-training. By these means, our methods not only bridge the gap between pretrain-finetune, but also enhance PLMs to better capture the alignment between different languages. Extensive experiments prove that our method achieves clearly superior results on multiple xSL benchmarks with limited pretraining data. Our methods also surpass the previous state-of-the-art methods by a large margin in few-shot data settings, where only a few hundred training examples are available.

show abstract

ReasonBERT: Pre-trained to Reason with Distant Supervision

Cited by 11 publications

References 23 publications

Active Prompting with Chain-of-Thought for Large Language Models

Active Prompting with Chain-of-Thought for Large Language Models

Efficient Training of Language Models to Fill in the Middle

Bridging the Gap between Language Models and Cross-Lingual Sequence Labeling

Contact Info

Product

Resources

About