A Survey in Mathematical Language Processing

Jordan, Meadows,; Freitas, André Victor Lucci

doi:10.48550/arxiv.2205.15231

Cited by 3 publications

(2 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent literature, Meadows et al [23] and Lu et al [24] conducted comprehensive surveys on the emerging deep learning-based models developed for solving math word problems. These studies systematically classify and document the network architectures and training techniques utilized by these models.…”

Section: The Previous Survey Workmentioning

confidence: 99%

“…Zhang et al [1] classified and analyzed different representation learning methods according to technical characteristics. Meadows et al [23] and Lu et al [24] conducted a literature review on the recent deep learning-based models for solving math word problems. Lan et al [10] established a unified algorithm test platform and conducted comparative experiments on typical neural solvers.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Comparative study of typical neural solvers in solving math word problems

He,

Yu,

Huang

et al. 2024

Complex Intell. Syst.

View full text Add to dashboard Cite

In recent years, there has been a significant increase in the design of neural network models for solving math word problems (MWPs). These neural solvers have been designed with various architectures and evaluated on diverse datasets, posing challenges in fair and effective performance evaluation. This paper presents a comparative study of representative neural solvers, aiming to elucidate their technical features and performance variations in solving different types of MWPs. Firstly, an in-depth technical analysis is conducted from the initial deep neural solver DNS to the state-of-the-art GPT-4. To enhance the technical analysis, a unified framework is introduced, which comprises highly reusable modules decoupled from existing MWP solvers. Subsequently, a testbed is established to conveniently reproduce existing solvers and develop new solvers by combing these reusable modules, and finely regrouped datasets are provided to facilitate the comparative evaluation of the designed solvers. Then, comprehensive testing is conducted and detailed results for eight representative MWP solvers on five finely regrouped datasets are reported. The comparative analysis yields several key findings: (1) Pre-trained language model-based solvers demonstrate significant accuracy advantages across nearly all datasets, although they suffer from limitations in math equation calculation. (2) Models integrated with tree decoders exhibit strong performance in generating complex math equations. (3) Identifying and appropriately representing implicit knowledge hidden in problem texts is crucial for improving the accuracy of math equation generation. Finally, the paper also discusses the major technical challenges and potential research directions in this field. The insights gained from this analysis offer valuable guidance for future research, model development, and performance optimization in the field of math word problem solving.

show abstract

Section: The Previous Survey Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Comparative study of typical neural solvers in solving math word problems

He,

Yu,

Huang

et al. 2024

Complex Intell. Syst.

View full text Add to dashboard Cite

show abstract

Towards AI-Assisted Synthesis of Verified Dafny Methods

Misu,

Lopes,

et al. 2024

Proc. ACM Softw. Eng.

View full text Add to dashboard Cite

Large language models show great promise in many domains, including programming. A promise is easy to make but hard to keep, and language models often fail to keep their promises, generating erroneous code. A promising avenue to keep models honest is to incorporate formal verification: generating programs’ specifications as well as code so that the code can be proved correct with respect to the specifications. Unfortunately, existing large language models show a severe lack of proficiency in verified programming. In this paper, we demonstrate how to improve two pretrained models’ proficiency in the Dafny verification-aware language. Using 178 problems from the MBPP dataset, we prompt two contemporary models (GPT-4 and PaLM-2) to synthesize Dafny methods. We use three different types of prompts: a direct Contextless prompt; a Signature prompt that includes a method signature and test cases, and a Chain of Thought (CoT) prompt that decomposes the problem into steps and includes retrieval augmentation generated example problems and solutions. Our results show that GPT-4 performs better than PaLM-2 on these tasks and that both models perform best with the retrieval augmentation generated CoT prompt. GPT-4 was able to generate verified, human-evaluated, Dafny methods for 58% of the problems, however, GPT-4 managed only 19% of the problems with the Contextless prompt, and even fewer (10%) for the Signature prompt. We are thus able to contribute 153 verified Dafny solutions to MBPP problems, 50 that we wrote manually, and 103 synthesized by GPT-4. Our results demonstrate that the benefits of formal program verification are now within reach of code generating large language models. Likewise, program verification systems can benefit from large language models, whether to synthesize code wholesale, to generate specifications, or to act as a "programmer’s verification apprentice", to construct annotations such as loop invariants which are hard for programmers to write or verification tools to find. Finally, we expect that the approach we have pioneered here — generating candidate solutions that are subsequently formally checked for correctness — should transfer to other domains (e.g., legal arguments, transport signaling, structural engineering) where solutions must be correct, where that correctness must be demonstrated, explained and understood by designers and end-users.

show abstract

Automatic Programming vs. Artificial Intelligence

Noble

2024

Proceedings of the 1st ACM International Conference on AI-Powered Software

View full text Add to dashboard Cite

A Survey in Mathematical Language Processing

Cited by 3 publications

References 50 publications

Comparative study of typical neural solvers in solving math word problems

Comparative study of typical neural solvers in solving math word problems

Towards AI-Assisted Synthesis of Verified Dafny Methods

Automatic Programming vs. Artificial Intelligence

Contact Info

Product

Resources

About