PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics

Abstract

In order for language models to aid physics research, they must first encode representations of mathematical and natural language discourse which lead to coherent explanations, with correct ordering and relevance of statements. We present a collection of datasets developed to evaluate the performance of language models in this regard, which measure capabilities with respect to sentence ordering, position, section prediction, and discourse coherence. Analysis of the data reveals equations and sub-disciplines which are most common in physics discourse, as well as the sentence-level frequency of equations and expressions. We present baselines which demonstrate how contemporary language models are challenged by coherence related tasks in physics, even when trained on mathematical natural language objectives.

A Survey in Mathematical Language Processing

“…Communicating quantitative science occurs through the medium of mathematical text, which contains expressions, formulae, and equations, most of which requiring accompanying description. Formulae and their explanations interweave with non-mathematical language to form cohesive discourse (Meadows et al, 2022). Approaches that consider mathematical text have been proposed to solve a number of related tasks, but are yet to surpass human-level performance.…”