Dict-BERT: Enhancing Language Model Pre-training with Dictionary

Abstract

Pre-trained language models (PLMs) aim to learn universal language representations by conducting self-supervised training tasks on largescale corpus. Since PLMs capture word semantics in different contexts, the quality of word representations highly depends on word frequency, which usually follows a heavy-tailed distribution in the pre-training corpus. Thus, the embeddings of rare words on the tail are usually poorly optimized. In this work, we focus on enhancing language model pre-training by leveraging definitions of the rare words in dictionary. To incorporate a rare word definition as a part of input, we fetch it from the dictionary and append it to the end of the input text sequence. In addition to training with the masked language modeling objective, we propose two novel self-supervised pre-training tasks on word-level and sentence-level alignment between the input text and rare word definition to enhance language representations. We evaluate the proposed model named Dict-BERT on the GLUE benchmark and eight specialized domain datasets. Extensive experiments show that Dict-BERT significantly improves the understanding of rare words and boosts model performance on various NLP downstream tasks.

Medicine-Engineering Interdisciplinary Research Based on Bibliometric Analysis: A Case Study on Medicine-Engineering Institutional Cooperation of Shanghai Jiao Tong University

“…Another research line aims to boost the performance of large-scale LLMs to higher levels by leveraging the self-generated content. Some works use the self-generation as contexts to assist themselves in answering questions, such as eliciting intermediate rationales as CoT (Kojima et al, 2023) or generating background articles for reading comprehension (Yu et al, 2023). While others instruct LLMs to generate demonstrations for ICL during inference , such as prompting LLMs to generate reliable QA pairs as self-prompted in-context demonstrations .…”

Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts

“…Incorporating external knowledge is essential for many NLG tasks to augment the limited textual information (Yu et al, 2022c;Dong et al, 2021;Yu et al, 2022b). Some recent work explored using graph neural networks (GNN) to reason over multihop relational knowledge graph (KG) paths (Zhou et al, 2018;Jiang et al, 2019;Zhang et al, 2020a;Wu et al, 2020;Yu et al, 2022a;Zeng et al, 2021).…”

Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions

“…Finally, we devise a denoising auto-encoder-style learning objective and train the network to reconstruct selective masked sentence parts. Our use of symbolic knowledge (Yu et al, 2021) of IEs to aid the learning of their embeddings results in the model needing a significantly small amount of data (∼60MB) compared to that required for LM pre-training (∼160GB of text for BART).…”