Contextualized Scene Imagination for Generative Commonsense Reasoning

Abstract

Humans use natural language to compose common concepts from their environment into plausible, day-to-day scene descriptions. However, such generative commonsense reasoning (GCSR) skills are lacking in state-of-the-art text generation methods. Descriptive sentences about arbitrary concepts generated by neural text generation models (e.g., pre-trained text-to-text Transformers) are often grammatically fluent but may not correspond to human common sense, largely due to their lack of mechanisms to capture concept relations, to identify implicit concepts, and to perform generalizable reasoning about unseen concept compositions. In this paper, we propose an Imagine-and-Verbalize (I&V) method, which learns to imagine a relational scene knowledge graph (SKG) with relations between the input concepts, and leverage the SKG as a constraint when generating a plausible scene description. We collect and harmonize a set of knowledge resources from different domains and modalities, providing a rich auxiliary supervision signal for I&V. The experiments demonstrate the effectiveness of I&V in improving language models on both concept-to-sentence and concept-to-story generation tasks, while enabling the model to learn well from fewer task examples and generate SKGs that make common sense to human annotators 1 .

CoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations

“…For future work, we would sample sub-graph structures to explore more meaningful event-centric commonsense knowledge (Wang et al, 2021a). Moreover, we will equip our models with generative abilities by finetuning powerful T5 (Raffel et al, 2020) or BART models to help narrative story completion (Ji et al, 2020), commonsense inference (Gabriel et al, 2021), event infilling tasks (Lin et al, 2021).…”