Natural Language Deduction through Search over Statement Compositions

Bostrom, Kaj; Sprague, Zayne; Chaudhuri, Swarat; Durrett, Greg

doi:10.48550/arxiv.2201.06028

Cited by 3 publications

(15 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 Middle). In contrast, stepwise methods generate the proof as individual proof steps, either forward Sanyal et al, 2022;Bostrom et al, 2022) or backward (Dalvi et al, 2022). Our method generates proofs stepwise, in the forward direction.…”

Section: Related Workmentioning

confidence: 99%

“…When generating a proof step, prior work has observed that if the hypothesis is available, the model often uses it to hallucinate the intermediate conclusion instead of drawing valid logical inferences (Table 3). Therefore, ProofWriter FaiRR (Sanyal et al, 2022), and SCSearch (Bostrom et al, 2022) explicitly ban the model from accessing the hypothesis when generating intermediate conclusions, forcing it to draw inference from known premises only. However, without the hypothesis, the model may generate many valid proof steps irrelevant to the hypothesis.…”

Section: Related Workmentioning

confidence: 99%

“…Table 1: A comparison of our NLProofS with prior methods for proof generation: PRover (Saha et al, 2020), Entail-mentWriter , ProofWriter , FaiRR (Sanyal et al, 2022), SCSearch (Bostrom et al, 2022), and a concurrent work (Dalvi et al, 2022). Dalvi et al (2022) is a concurrent work also using a verifier in generating proof steps.…”

Section: Nlproofs (Ours) →mentioning

confidence: 99%

“…The proof tree may contain intermediate conclusions, which need to be generated by the model. Existing methods generate the proof either in a single shot (Saha et al, 2020; or step by step (Sanyal et al, 2022;Bostrom et al, 2022;Dalvi et al, 2022). Stepwise methods leverage the compositionality of proofs, making it easier for the model to learn and generalize to long proofs .…”

Section: Introductionmentioning

confidence: 99%

“…However, existing stepwise methods suffer from a trade-off between generating valid steps and relevant steps. Prior works (Sanyal et al, 2022;Bostrom et al, 2022) have observed that, given the hypothesis, the model often learns to hallucinate invalid proof steps leading to the hypothesis, instead of performing valid logical inference (example in Table 3). To mitigate this issue, previous attempts have restricted the model from accessing the hypothesis, forcing it to generate conclusions based solely on known premises.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Generating Natural Language Proofs with Verifier-Guided Search

Yang¹,

Jia²,

Chen³

2022

Preprint

View full text Add to dashboard Cite

Deductive reasoning-drawing conclusions from assumptions-is a challenging problem in NLP. In this work, we focus on proof generation: given a hypothesis and a set of supporting facts in natural language, the model generates a proof tree indicating how to deduce the hypothesis from supporting facts. Instead of generating the entire proof in one shot, prior work has demonstrated the promise of stepwise generation but achieved limited success on real-world data. Existing stepwise methods struggle to generate proof steps that are both valid and relevant. In this paper, we present a novel stepwise method NLProofS (Natural Language Proof Search), which learns to generate relevant steps conditioning on the hypothesis. At the core of our approach, we train an independent verifier to check the validity of proof steps. Instead of generating steps greedily, we search for proofs maximizing a global proof score judged by the verifier. NLProofS achieves state-of-the-art performance on En-tailmentBank and RuleTaker. For example, it improves the percentage of correctly predicted proofs from 20.9% to 33.3% in the distractor setting of EntailmentBank. This is the first time stepwise methods have led to better generation of challenging human-authored proofs. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Nlproofs (Ours) →mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Generating Natural Language Proofs with Verifier-Guided Search

Yang¹,

Jia²,

Chen³

2022

Preprint

View full text Add to dashboard Cite

show abstract

Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model

Eisenstein¹,

Andor²,

Bohnet³

et al. 2022

Preprint

View full text Add to dashboard Cite

Explainable question answering systems should produce not only accurate answers but also rationales that justify their reasoning and allow humans to check their work. But what sorts of rationales are useful and how can we train systems to produce them? We propose a new style of rationale for open-book question answering, called markup-and-mask, which combines aspects of extractive and free-text explanations. In the markup phase, the passage is augmented with free-text markup that enables each sentence to stand on its own outside the discourse context. In the masking phase, a sub-span of the marked-up passage is selected. To train a system to produce markup-and-mask rationales without annotations, we leverage in-context learning. Specifically, we generate silver annotated data by sending a series of prompts to a frozen pretrained language model, which acts as a teacher. We then fine-tune a smaller student model by training on the subset of rationales that led to correct answers. The student is "honest" in the sense that it is a pipeline: the rationale acts as a bottleneck between the passage and the answer, while the "untrusted" teacher operates under no such constraints. Thus, we offer a new way to build trustworthy pipeline systems from a combination of end-task annotations and frozen pretrained language models.

show abstract

Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering

Liu¹,

Skyler²,

Lu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Knowledge underpins reasoning. Recent research demonstrates that when relevant knowledge is provided as additional context to commonsense question answering (QA), it can substantially enhance the performance even on top of state-of-the-art. The fundamental challenge is where and how to find such knowledge that is high quality and on point with respect to the question; knowledge retrieved from knowledge bases are incomplete and knowledge generated from language models are inconsistent.We present RAINIER 1 , or Reinforced Knowledge Introspector, that learns to generate contextually relevant knowledge in response to given questions. Our approach starts by imitating knowledge generated by GPT-3, then learns to generate its own knowledge via reinforcement learning where rewards are shaped based on the increased performance on the resulting question answering. RAINIER demonstrates substantial and consistent performance gains when tested over 9 different commonsense benchmarks: including 5 datasets that are seen during model training, as well as 4 datasets that are kept unseen. Our work is the first to report that knowledge generated by models that are orders of magnitude smaller than GPT-3, even without direct supervision on the knowledge itself, can exceed the quality of commonsense knowledge elicited from GPT-3.

show abstract

Natural Language Deduction through Search over Statement Compositions

Cited by 3 publications

References 19 publications

Generating Natural Language Proofs with Verifier-Guided Search

Generating Natural Language Proofs with Verifier-Guided Search

Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model

Rainier: Reinforced Knowledge Introspector for Commonsense Question Answering

Contact Info

Product

Resources

About