Symbolic Knowledge Distillation: from General Language Models to Commonsense Models

Abstract

The common practice for training commonsense models has gone from-human-to-corpusto-machine: humans author commonsense knowledge graphs in order to train commonsense models. In this work, we investigate an alternative, from-machine-to-corpus-tomachine: general language models author these commonsense knowledge graphs to train commonsense models.Our study leads to a new framework, Symbolic Knowledge Distillation. As with prior art in Knowledge Distillation (Hinton et al., 2015), our approach uses larger models to teach smaller models. A key difference is that we distill knowledge symbolically-as text-in addition to the resulting neural model. We distill only one aspect-the commonsense of a general language model teacher, allowing the student to be a different type of model, a commonsense model. Altogether, we show that careful prompt engineering and a separately trained critic model allow us to selectively distill highquality causal commonsense from GPT-3, a general language model. Empirical results demonstrate that, for the first time, a human-authored commonsense knowledge graph is surpassed by our automatically distilled variant in all three criteria: quantity, quality, and diversity. In addition, it results in a neural commonsense model that surpasses the teacher model's commonsense capabilities despite its 100x smaller size. We apply this to the ATOMIC resource, and will share our new symbolic knowledge graph and commonsense models 1 .

What happens before and after: Multi-Event Commonsense in Event Coreference Resolution

“…Recent work augmented datasets by fine-tuning a pre-trained LM on real data, then generated new, silver-labelled instances (Anaby-Tavor et al, 2020;Papanikolaou and Pierleoni, 2020;Kumar et al, 2020). Similarly, the few-shot capabilities of GPT-3 (Brown et al, 2020) were leveraged to generate free-text explanations (Wiegreffe et al, 2022), semantically-related sentence pairs (Schick and Schütze, 2021), atomic event commonsense triples (West et al, 2022), and labels for various generation and understanding tasks . In this work, we finetune GPT-3 with minimal human supervision to generate additional contextual data pertaining to events.…”

KAT: A Knowledge Augmented Transformer for Vision-and-Language

“…Recent approaches have shown a great potential to incorporate external knowledge for knowledgebased VQA. Several methods explore aggregating the external knowledge either in the form of structured knowledge graphs (Garderes et al, 2020;Narasimhan et al, 2018;Li et al, 2020b;Wang et al, 2017a,b), unstructured knowledge bases (Marino et al, 2021;Wu et al, 2022;Luo et al, 2021), and neural-symbolic inference based knowledge (Chen et al, 2020;West et al, 2021). In these methods, object detectors (Ren et al, 2015) and scene classifiers (He et al, 2016) are used to associate images with external knowledge.…”

Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge

“…Commonsense knowledge acquisition is a longstanding challenge in natural language processing (Charniak, 1973;Hwang et al, 2021;Zhang et al, 2021), and current approaches rely on knowledge acquired by pre-trained Transformer language models (Bosselut et al, 2019;Zhang et al, 2020;West et al, 2021). The commonsense reasoning ability of these language models has been evaluated using behavioral probes (Ettinger, 2020;Misra et al, 2021;He et al, 2021) and downstream, fine-tuned evaluations (Banerjee et al, 2021;Zhou et al, 2021;.…”