RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder

RetroMAE v2: Duplex Masked Auto-Encoder For Pre-Training Retrieval-Oriented Language Models

“…Later on, the auto-encoding based pre-training algorithms receive growing interests: the input sentences are encoded into embeddings and reconstructed back to the original sentences (Lu et al, 2021;. The recently proposed methods, such as SimLM and RetroMAE (Xiao et al, 2022a), extend the previous auto-encoding framework by upgrading the encoding and decoding mechanisms, which substantially improves the quality of deep semantic retrieval.…”

“…Later on, auto-encoding is found to be more effective Lu et al, 2021), where the language models are learned to reconstruct the input based on the generated embeddings. The recent work RetroMAE (Xiao et al, 2022a) extends the previous auto-encoding methods by introducing the enhanced encoding and decoding mechanisms, which leads to remarkable improvements on general retrieval benchmarks.…”

“…The existing retrieval-oriented pre-trained models mainly rely on the contextualized embedding from the special token, i.e., [CLS], to represent the semantic about input (Gao and Callan, 2021;Lu et al, 2021;Xiao et al, 2022a;. However, recent study finds that other ordinary tokens may provide extra information and help to generate better semantic representations (Lin et al, 2022).…”

“…It introduces two decoding modules, which work together to enhance the semantic representation capacity for both types of contextualized embeddings. Particularly, we leverage the decoder from RetroMAE (Xiao et al, 2022a), where the [CLS] embedding, joined with the masked input, is used to recover the original sentence via an onelayer transformer. Meanwhile, the contextualized embeddings from ordinary tokens are transformed into the vocabulary space (i.e, |V |-dim vectors) via a linear projection unit (i.e., a d × |V | matrix).…”

“…

To better support retrieval applications such as web search and question answering, growing effort is made to develop retrieval-oriented language models (Gao and Callan, 2021;Xiao et al, 2022a). Most of the existing works focus on improving the semantic representation capability for the contextualized embedding of [CLS] token.

…”