Students Who Study Together Learn Better: On the Importance of Collective Knowledge Distillation for Domain Transfer in Fact Verification

Abstract

While neural networks produce state-of-the-art performance in several NLP tasks, they depend heavily on lexicalized information, which transfers poorly between domains. Previous work (Suntwal et al., 2019) proposed delexicalization as a form of knowledge distillation to reduce dependency on such lexical artifacts. However, a critical unsolved issue that remains is how much delexicalization should be applied? A little helps reduce over-fitting, but too much discards useful information. We propose Group Learning (GL), a knowledge and model distillation approach for fact verification. In our method, while multiple student models have access to different delexicalized data views, they are encouraged to independently learn from each other through pair-wise consistency losses. In several cross-domain experiments between the FEVER and FNC fact verification datasets, we show that our approach learns the best delexicalization strategy for the given training dataset and outperforms state-of-theart classifiers that rely on the original data. Claim EvidencePlain text J. R. R. Tolkien created Gimli . A dwarf warrior , he is the son of Glóin -LRB-a character from Tolkien's earlier novel, The Hobbit -RRB-. Gimli is a fictional character from J. R. R. Tolkien s Middle-earth legendarium, featured in The Lord of the Rings .OA-NER personC1 created personC2. A dwarf warrior, he is the son of personE1 -LRB-a character from per-sonC1's earlier novel, The Hobbit -RRB-. personC2 is a fictional character from personC1's locationE1 legendarium, featured in The Lord of the Rings.FIGER Specific authorC1 created locationC1. A dwarf warrior, he is the son of personE1 -LRB-a character from authorE1's earlier novel, The Hobbit -RRB-. locationC1 is a fictional character from authorC1's written_workE1 legendarium, featured in The Lord of the Rings.