<i>Break, Perturb, Build</i>: Automatic Perturbation of Reasoning Paths Through Question Decomposition

Geva, Mor; Wolfson, Tomer; Berant, Jonathan

doi:10.1162/tacl_a_00450

Cited by 9 publications

(10 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…6 For IIRC, we consider two settings: gold-setting (IIRC-G) which uses only gold supporting sentences as reading comprehension context, and retrieved-setting (IIRC-R) which retrieves paragraphs using a retrieval marginalization method (Ni et al, 2021). We evaluate robustness using DROP contrast set and DROP BPB contrast set (Geva et al, 2022) 7 . For robustness evaluation, we only fine-tune on DROP dataset and evaluate on the contrast sets directly.…”

Section: Datasetsmentioning

confidence: 99%

“…The BREAK dataset (Wolfson et al, 2020) on the other hand, defined a standardized meaning representation format (inspired by semantic parsing) for several QA datasets. This shared representation has allowed the development of contrastive datasets (Geva et al, 2022). In this work, we leverage these annotations to build a dataset to teach broad reasoning skills to models.…”

Section: Related Workmentioning

confidence: 99%

“…The closest work in this line is the Break-Perturb-Build dataset (Geva et al, 2022). BPB dataset also used QDMR but with the goal of creating contrastive questions via minor question perturbation (Kaushik et al, 2019;.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Teaching Broad Reasoning Skills via Decomposition-Guided Contexts

Trivedi¹,

Balasubramanian²,

Khot³

et al. 2022

Preprint

View full text Add to dashboard Cite

Question-answering datasets require a broad set of reasoning skills. We show how to use question decompositions to teach language models these broad reasoning skills in a robust fashion. Specifically, we use widely available QDMR representations to programmatically create synthetic contexts for real questions in six multihop reasoning datasets. These contexts are carefully designed to avoid common reasoning shortcuts prevalent in real contexts that prevent models from learning the right skills. This results in a pretraining dataset, named TeaBReaC, containing 525K multihop questions (with associated formal programs) covering about 900 reasoning patterns. We show that pretraining standard language models (LMs) on TeaBReaC before fine-tuning them on target datasets improves their performance by up to 13 EM points across 3 multihop QA datasets, with a 30 point gain on more complex questions. The resulting models also demonstrate higher robustness, with a 6-11 point improvement on two contrast sets. Furthermore, TeaBReaC pretraining substantially improves model performance and robustness even when starting with numeracy-aware LMs pretrained using recent methods (e.g., PReasM). Our work thus shows how one can effectively use decomposition-guided contexts to robustly teach multihop reasoning. 1

show abstract

Section: Datasetsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Teaching Broad Reasoning Skills via Decomposition-Guided Contexts

Trivedi¹,

Balasubramanian²,

Khot³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…We consult quantifier taxonomy studies (Keenan and Westerståhl, 1997;Peters and Westerståhl, 2006;Szymanik and Thorne, 2015;Szymanik, 2016) and derive a categorization set for quantifier analysis in NLU benchmarks. In coreference (Ogrodniczuk et al, 2019, 2020), negation (Hossain et al, 2020Hartmann et al, 2021), and consistency (Li et al, 2019;Ribeiro et al, 2019;Asai and Hajishirzi, 2020;Geva et al, 2022)-there has been little work on generalized quantifiers as a source of error in NLU, let alone in multilingual NLU. It remains an open problem whether LMs represent the semantics of quantifiers words adequately, or if they provide a basis for resolving scopal ambiguities.…”

Section: Introductionmentioning

confidence: 99%

Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks

Cui¹,

Hershcovich²,

Søgaard³

2022

Preprint

View full text Add to dashboard Cite

Logical approaches to representing language have developed and evaluated computational models of quantifier words since the 19th century, but today's NLU models still struggle to capture their semantics. We rely on Generalized Quantifier Theory for languageindependent representations of the semantics of quantifier words, to quantify their contribution to the errors of NLU models. We find that quantifiers are pervasive in NLU benchmarks, and their occurrence at test time is associated with performance drops. Multilingual models also exhibit unsatisfying quantifier reasoning abilities, but not necessarily worse for non-English languages. To facilitate directlytargeted probing, we present an adversarial generalized quantifier NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quantifier reasoning.

show abstract

“…As QDMR is derived entirely from the original question, it is agnostic to the underlying domain, schema or even the form of knowledge representation. It has been applied to questions on text and images and relational databases [16,46,52]. Here, we utilize QDMR structure and show that it can successfully be mapped to SQL.…”

mentioning

confidence: 99%

Weakly Supervised Text-to-SQL Parsing through Question Decomposition

Wolfson¹,

Deutch²,

Berant³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Natural Language Interfaces to Databases (NLIDBs), where users pose queries in Natural Language (NL), are crucial for enabling non-experts to gain insights from data. Developing such interfaces, by contrast, is dependent on experts who often code heuristics for mapping NL to SQL. Alternatively, NLIDBs based on machine learning models rely on supervised examples of NL to SQL mappings (NL-SQL pairs) used as training data. Such examples are again procured using experts, which typically involves more than a oneoff interaction. Namely, each data domain in which the NLIDB is deployed may have different characteristics and therefore require either dedicated heuristics or domain-specific training examples.To this end, we propose an alternative approach for training machine learning-based NLIDBs, using weak supervision. We use the recently proposed question decomposition representation called QDMR, an intermediate between NL and formal query languages. Recent work has shown that non-experts are generally successful in translating NL to QDMR. We consequently use NL-QDMR pairs, along with the question answers, as supervision for automatically synthesizing SQL queries. The NL questions and synthesized SQL are then used to train NL-to-SQL models, which we test on five benchmark datasets. Extensive experiments show that our solution, requiring zero expert annotations, performs competitively with models trained on expert annotated data.

show abstract

Break, Perturb, Build: Automatic Perturbation of Reasoning Paths Through Question Decomposition

Cited by 9 publications

References 33 publications

Teaching Broad Reasoning Skills via Decomposition-Guided Contexts

Teaching Broad Reasoning Skills via Decomposition-Guided Contexts

Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks

Weakly Supervised Text-to-SQL Parsing through Question Decomposition

Contact Info

Product

Resources

About