MathPrompter: Mathematical Reasoning using Large Language Models

Abstract

Large Language Models (LLMs) have limited performance when solving arithmetic reasoning tasks and often provide incorrect answers. Unlike natural language understanding, math problems typically have a single correct answer, making the task of generating accurate solutions more challenging for LLMs. To the best of our knowledge, we are not aware of any LLMs that indicate their level of confidence in their responses which fuels a trust deficit in these models impeding their adoption. To address this deficiency, we propose 'MathPrompter', a technique that improves performance of LLMs on arithmetic problems along with increased reliance in the predictions. MathPrompter uses the Zero-shot chain-of-thought prompting technique to generate multiple Algebraic expressions or Python functions to solve the same math problem in different ways and thereby raise the confidence level in the output results. This is in contrast to other prompt based CoT methods, where there is no check on the validity of the intermediate steps followed. Our technique improves over state-of-the-art on the MultiArith dataset (78.7% → 92.5%) evaluated using 175B parameter GPT-based LLM.

Geography of Technology Transfer in China

“…This approach successfully solves, explains and generates math problems at the university level. MathPrompter [66] employs a zero-shot chainof-thought prompting technique to generate multiple algebraic expressions or Python functions in varied ways to solve the same math problem, enhancing reliance on output results. PAL [44] introduces an innovative approach to bolster the performance of pre-trained language models (PLMs) in mathematical problem-solving.…”

“…Recent research has highlighted the growing capabilities of LLMs in the field of mathematical word problem solving, emphasizing the trend toward more nuanced and sophisticated AI-driven mathematical analysis. MathPrompter [66] uses LLM called GPT3 DaVinci to solve MWPs with excellent results, demonstrating the potential of LLMs to not only explain but also generate complex mathematical reasoning, reflecting a human-like understanding of complex problem sets.…”

Red Teaming for Large Language Models At Scale: Tackling Hallucinations on Mathematics Tasks

“…Previous work, such as that of Frieder et al (2023), has shown that advanced LLMs -specifically ChatGPT -tend to be highly inconsistent on mathematics tasks. Similarly, Imani et al (2023) found that hallucinations tend to be amplified when models attempt mathematical reasoning. We believe that equations and puzzles are useful testing grounds because the quality of the model's answers can be objectively evaluated.…”

Generalized Planning in PDDL Domains with Pretrained Large Language Models

“…CoT summarization is related to several techniques that ask the LLM to outline its "thinking" before arriving at a final implementation (Wei et al 2022;Jiang et al 2023;Zheng et al 2023). A number of recent works also use programs as prompts (i.e., a structured chain of thought) in an attempt to help LLMs perform mathematical reasoning (Gao et al 2022;Imani, Du, and Shrivastava 2023). Related to our automated debugging, Xia and Zhang 2023a) consider a related paradigm, but where feedback comes from humans, rather than automated checks.…”