Time-Stamped Language Model: Teaching Language Models to Understand The Flow of Events

Abstract

Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process. Firstly, we propose to formulate this task as a question answering problem. This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding. Secondly, since the transformerbased language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model (TSLM model) to encode event information in LMs architecture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published stateof-the-art results with a 3.1% increase in F1 score. Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general.

The Role of Semantic Parsing in Understanding Procedural Text

“…This paper primarily focuses on tracking entities' states and properties throughout a procedural text. Recent research has addressed this problem by predicting actions and properties on local context (Prolocal) , autoregressive global predictions based on distance vectors (Proglobal) , integrating structural common-sense knowledge built over VerbNet (ProStruct) , building dynamic knowledge graphs over entities (KG-MRC) (Das et al, 2018), explicitly encoding the model to explain dependencies between actions (XPAD) , formulating local predictions and global sequential information flow and sequential constraints (NCET) (Gupta and Durrett, 2019), formulating the task in a QA setting (DynaPro, TSLM) (Amini et al, 2020;Faghihi and Kordjamshidi, 2021), inte-grating common-sense knowledge from Concetp-Net (KOALA) , utilizing large generative language models (LEMON) (Shi et al, 2022), or using both the question answering setting and sequential structural constraints at the same time (CGLI) (Ma et al, 2022). All the models mentioned above investigate different neural architectures to tackle the task, while we are more interested in augmenting them with additional knowledge from semantic parsers.…”

“…Except for the common-sense and the ability to make consistent global decisions actions (Gupta and Durrett, 2019), the other challenges might have only been indirectly tackled in the recent research (Huang et al, 2021;Faghihi and Kordjamshidi, 2021), but have neither been addressed explicitly nor properly evalu-ated to measure their success on resolving these challenges. In this paper, we evaluate whether semantic parsers can alleviate some of these challenges.…”

LEMON: Language-Based Environment Manipulation via Execution-Guided Pre-training

“…Second, they either rely on natural language-program pairs as supervision or require complex heuristic rules, which is costly. Recent approaches generally treat the LEM task as a state prediction problem by predicting the goal state directly Du et al, 2019;Das et al, 2019;Rajaby Faghihi and Kordjamshidi, 2021;Zhang et al, 2021). These models can eliminate the data collection issue, but design complex models to encode environment states instead, to meet the need of different kinds of environments.…”

Transferring Procedural Knowledge Across Commonsense Tasks

“…Building AI agents that understand stories is central to many domains, ranging from cooking [28] to science [12]. This is because practically any situation can be associated with a story that requires an agent to judge and explain its plausibility [7].…”

“…The requirement of benchmark-specific training also limits their generalization to novel benchmarks and tasks. A parallel stream of research [28,21] * Email: {yifjia,ilievski}@isi.edu. kaixinm@andrew.cmu.edu.…”