Towards General Natural Language Understanding with Probabilistic
                    Worldbuilding

Saparov, Abulhair; Mitchell, Tom M.

doi:10.1162/tacl_a_00463

Cited by 4 publications

(4 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One other aspect of LLMs that may limit them when applied to reasoning is that they produce natural language text, which may be ambiguous and diverse. These considerations motivate us to study MSPs as structured representations of meaning, which can in turn be used to generate natural language (Saparov and Mitchell, 2022).…”

Section: Related Workmentioning

confidence: 99%

World Models for Math Story Problems

Opedal¹,

Stoehr²,

Saparov³

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

Solving math story problems is a complex task for students and NLP models alike, requiring them to understand the world as described in the story and reason over it to compute an answer. Recent years have seen impressive performance on automatically solving these problems with large pre-trained language models and innovative techniques to prompt them. However, it remains unclear if these models possess accurate representations of mathematical concepts. This leads to lack of interpretability and trustworthiness which impedes their usefulness in various applications. In this paper, we consolidate previous work on categorizing and representing math story problems and develop MATHWORLD, which is a graph-based semantic formalism specific for the domain of math story problems. With MATHWORLD, we can assign world models to math story problems which represent the situations and actions introduced in the text and their mathematical relationships. We combine math story problems from several existing datasets and annotate a corpus of 1, 019 problems and 3, 204 logical forms with MATHWORLD. Using this data, we demonstrate the following use cases of MATHWORLD: (1) prompting language models with synthetically generated questionanswer pairs to probe their reasoning and world modeling abilities, and (2) generating new problems by using the world models as a design space. https://github.com/eth-nlped/ mathworld Arithmetic coverage Conceptual coverage Semantic granularity Annotations? Mapping to formal logic? MATHWORLD (+, −, ×, ÷) Transfer Rate Comparison Part-whole World model Yes Yes Hosseini et al. (2014) (+, −) Transfer World model No No Mitra and Baral (2016) (+, −) Transfer Comparison (add) Part-whole Concepts & equations Yes No Roy and Roth (2018) (+, −, ×, ÷) Transfer Rate Comparison Part-whole Concepts & equations No No

show abstract

Section: Related Workmentioning

confidence: 99%

World Models for Math Story Problems

Opedal¹,

Stoehr²,

Saparov³

et al. 2023

Findings of the Association for Computational Linguistics: ACL 2023

View full text Add to dashboard Cite

show abstract

“…There are two broad research approaches for reasoning in NLP: (1) reasoning over a formal symbolic language, possibly with neuro-symbolic methods and/or semantic parsing (Saparov & Mitchell, 2022;Kapanipathi et al, 2021;Dong et al, 2019;Rocktäschel & Riedel, 2017), or (2) reasoning directly over natural language (Chen et al, 2021;Bostrom et al, 2022;Welleck et al, 2021;Bhagavatula et al, 2020). While PRONTOQA is generated from symbolic ontologies, the examples themselves are in natural language, and so provides value to both research approaches.…”

Section: Related Workmentioning

confidence: 99%

Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

Saparov¹,

He²

2022

Preprint

View full text Add to dashboard Cite

Large language models (LLMs) have shown remarkable reasoning capabilities given chain-of-thought prompts (examples with intermediate reasoning steps). Existing benchmarks measure reasoning ability indirectly, by evaluating accuracy on downstream tasks such as mathematical reasoning. However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought. To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called PRONTOQA, where each example is generated from a synthetic world model represented in first-order logic. This allows us to parse the generated chain-ofthought into symbolic proofs for formal analysis. Our analysis on INSTRUCTGPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and so are generally capable of reasoning, even in fictional contexts. However, they have difficulty with proof planning: When multiple valid deduction steps are available, they are not able to systematically explore the different options.

show abstract

“…It suggests that people solve situated problems by elaborating a mental picture of the situation, including elements that may be peripheral to a specific answer, rather than by constructing a deductive proof from a few key facts to an answer (Byrne, 1991). Recently, Saparov and Mitchell (2022) tried creating an internal "mental model" using a set of axioms that deductively explain the observations. Other studies in AI have attempted to identify such elements by studying what questions people naturally ask when reading text (Ko et al, 2020) or viewing images (Mostafazadeh et al, 2016).…”

Section: Related Workmentioning

confidence: 99%

DREAM: Improving Situational QA by First Elaborating the Situation

Gu¹,

Mishra²,

Clark³

2022

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua

View full text Add to dashboard Cite

When people answer questions about a specific situation, e.g., "I cheated on my mid-term exam last week. Was that wrong?", cognitive science suggests that they form a mental picture of that situation before answering. While we do not know how language models (LMs) answer such questions, we conjecture that they may answer more accurately if they are also provided with additional details about the question situation, elaborating the "scene". To test this conjecture, we train a new model, DREAM, to answer questions that elaborate the scenes that situated questions are about, and then provide those elaborations as additional context to a question-answering (QA) model. We find that DREAM is able to create better scene elaborations (more accurate, useful, and consistent) than a representative state-of-the-art, zero-shot model (Macaw). We also find that using the scene elaborations as additional context improves the answer accuracy of a downstream QA system, including beyond that obtainable by simply further fine-tuning the QA system on DREAM's training data. These results suggest that adding focused elaborations about a situation can improve a system's reasoning about it, and may serve as an effective way of injecting new scenario-based knowledge into QA models. Finally, our approach is dataset-neutral; we observe improved QA performance across different models, with even bigger gains on models with fewer parameters. 1

show abstract

Towards General Natural Language Understanding with Probabilistic Worldbuilding

Cited by 4 publications

References 39 publications

World Models for Math Story Problems

World Models for Math Story Problems

Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

DREAM: Improving Situational QA by First Elaborating the Situation

Contact Info

Product

Resources

About