LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization

Abstract

Language model pre-training based on large corpora has achieved tremendous success in terms of constructing enriched contextual representations and has led to significant performance gains on a diverse range of Natural Language Understanding (NLU) tasks. Despite the success, most current pre-trained language models, such as BERT, are trained based on single-grained tokenization, usually with fine-grained characters or sub-words, making it hard for them to learn the precise meaning of coarse-grained words and phrases. In this paper, we propose a simple yet effective pretraining method named LICHEE to efficiently incorporate multi-grained information of input text. Our method can be applied to various pretrained language models and improve their representation capability. Extensive experiments conducted on CLUE and SuperGLUE demonstrate that our method achieves comprehensive improvements on a wide variety of NLU tasks in both Chinese and English with little extra inference cost incurred, and that our best ensemble model achieves the state-of-the-art performance on CLUE benchmark competition.

ERNIE-mmLayout: Multi-grained MultiModal Transformer for Document Understanding

“…On the contrary, we will get this information more easily if we look at them as a whole. The potential of such natural lexical units has been demonstrated in natural language models [12,46]. Similarly, as shown in Figure 1(c) and Figure 1(e), compared with image patches, salient visual regions can reflect richer semantic information.…”

“…However, existing layout-aware multimodal Transformers mainly focus on fine-grained information such as words and image patches. They ignore coarse-grained information including natural lexical units [12,46], like multi-word expressions and phrases, and salient visual regions [1,21], like attractive or prominent image regions. These coarse-grained elements contain high-density information and consistent semantics, which are valuable for document understanding.…”

MarkBERT: Marking Word Boundaries Improves Chinese BERT

“…The first category uses word information in the pretraining stage but represents a text as a sequence of characters when the pretrained model is applied to downstream tasks (Cui et al, 2019a;Lai et al, 2021). The second category uses word information when the pretrained model is used in downstream tasks (Su, 2020;Guo et al, 2021). In this paper, MarKBERT incorporate the boundary information in the training process latently.…”

“…-LICHEE (Guo et al, 2021): a multi-granularity Chinese pre-trained model that incorporates word and character representations at the embedding level.…”

“…There are several attempts at building Chinese BERT models where word information is considered. Existing studies tokenize a word as a basic unit (Su, 2020), as multiple characters (Cui et al, 2019a) or a combination of both Lai et al, 2021;Guo et al, 2021). However, due to the limit of the vocabulary size of BERT, these models only learn for a limited number (e.g., 40K) of words with high frequency.…”