While neural networks produce state-of-theart performance in several NLP tasks, they generally depend heavily on lexicalized information, which transfer poorly between domains. We present a combination of two strategies to mitigate this dependence on lexicalized information in fact verification tasks. We present a data distillation technique for delexicalization, which we then combine with a model distillation method to prevent aggressive data distillation. We show that by using our solution, not only does the performance of an existing state-of-the-art model remain at par with that of the model trained on a fully lexicalized data, but it also performs better than it when tested out of domain. We show that the technique we present encourages models to extract transferable facts from a given fact verification dataset.

No abstract

While neural networks produce state-of-the-art performance in several NLP tasks, they depend heavily on lexicalized information, which transfers poorly between domains. Previous work (Suntwal et al., 2019) proposed delexicalization as a form of knowledge distillation to reduce dependency on such lexical artifacts. However, a critical unsolved issue that remains is how much delexicalization should be applied? A little helps reduce over-fitting, but too much discards useful information. We propose Group Learning (GL), a knowledge and model distillation approach for fact verification. In our method, while multiple student models have access to different delexicalized data views, they are encouraged to independently learn from each other through pair-wise consistency losses. In several cross-domain experiments between the FEVER and FNC fact verification datasets, we show that our approach learns the best delexicalization strategy for the given training dataset and outperforms state-of-theart classifiers that rely on the original data. Claim EvidencePlain text J. R. R. Tolkien created Gimli . A dwarf warrior , he is the son of Glóin -LRB-a character from Tolkien's earlier novel, The Hobbit -RRB-. Gimli is a fictional character from J. R. R. Tolkien s Middle-earth legendarium, featured in The Lord of the Rings .OA-NER personC1 created personC2. A dwarf warrior, he is the son of personE1 -LRB-a character from per-sonC1's earlier novel, The Hobbit -RRB-. personC2 is a fictional character from personC1's locationE1 legendarium, featured in The Lord of the Rings.FIGER Specific authorC1 created locationC1. A dwarf warrior, he is the son of personE1 -LRB-a character from authorE1's earlier novel, The Hobbit -RRB-. locationC1 is a fictional character from authorC1's written_workE1 legendarium, featured in The Lord of the Rings.