An Effective Domain Adaptive Post-Training Method for BERT in Response Selection

Whang, Taesun; Lee, Dongyub; Lee, Chanhee; Yang, Kisu; Oh, Dongsuk; Lim, Heuiseok

doi:10.21437/interspeech.2020-2153

Cited by 72 publications

(89 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Post-training refers to the process of performing additional unsupervised training to a PLM such as BERT using unlabeled domain-specific data, prior to fine-tuning. It has been shown that this leads to improved performance by helping the PLM to adapt to the target domain [46][47][48][49]. We start with a monolingual PLM in L S and completely adapt it to L T .…”

Section: Transfer Learning As Post-trainingmentioning

confidence: 99%

Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models

Lee

Yang

Whang³

et al. 2021

Applied Sciences

Self Cite

View full text Add to dashboard Cite

Language model pretraining is an effective method for improving the performance of downstream natural language processing tasks. Even though language modeling is unsupervised and thus collecting data for it is relatively less expensive, it is still a challenging process for languages with limited resources. This results in great technological disparity between high- and low-resource languages for numerous downstream natural language processing tasks. In this paper, we aim to make this technology more accessible by enabling data efficient training of pretrained language models. It is achieved by formulating language modeling of low-resource languages as a domain adaptation task using transformer-based language models pretrained on corpora of high-resource languages. Our novel cross-lingual post-training approach selectively reuses parameters of the language model trained on a high-resource language and post-trains them while learning language-specific parameters in the low-resource language. We also propose implicit translation layers that can learn linguistic differences between languages at a sequence level. To evaluate our method, we post-train a RoBERTa model pretrained in English and conduct a case study for the Korean language. Quantitative results from intrinsic and extrinsic evaluations show that our method outperforms several massively multilingual and monolingual pretrained language models in most settings and improves the data efficiency by a factor of up to 32 compared to monolingual training.

show abstract

Section: Transfer Learning As Post-trainingmentioning

confidence: 99%

Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models

Lee

Yang

Whang³

et al. 2021

Applied Sciences

Self Cite

View full text Add to dashboard Cite

show abstract

“…Similar to BERT, we use special tokens like [CLS] to denote the beginning of the sequence, and [SEP] to separate the two modalities. Moreover, to inject the multi-turn dialog structure into the model, we utilize a special token [EOT] to denote end of turn (Whang et al, 2019), which informs the model when the dialog turn ends. As such, we prepare the input sequence into the format as…”

Section: Vision-dialog Transformer Encodermentioning

confidence: 99%

VD-BERT: A Unified Vision and Dialog Transformer with BERT

Wang¹,

Joty²,

Lyu³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Visual dialog is a challenging vision-language task, where a dialog agent needs to answer a series of questions through reasoning on the image content and dialog history. Prior work has mostly focused on various attention mechanisms to model such intricate interactions. By contrast, in this work, we propose VD-BERT, a simple yet effective framework of unified vision-dialog Transformer that leverages the pretrained BERT language models for Visual Dialog tasks. The model is unified in that ( 1) it captures all the interactions between the image and the multi-turn dialog using a single-stream Transformer encoder, and (2) it supports both answer ranking and answer generation seamlessly through the same architecture. More crucially, we adapt BERT for the effective fusion of vision and dialog contents via visually grounded training. Without the need of pretraining on external vision-language data, our model yields new state of the art, achieving the top position in both single-model and ensemble settings (74.54 and 75.35 NDCG scores) on the visual dialog leaderboard. Our code and pretrained models are released at https: //github.com/salesforce/VD-BERT.

show abstract

“…Baseline Models. Here we mainly introduce the state-of-the-art baseline: BERT-DPT (Whang et al, 2019), which fine-tunes BERT by optimizing the domain post-training (DPT) loss comprising both NSP and MLM objectives for response selection. Details of other baselines can be found in Appendix.…”

Section: Experiments Iv: Evaluation On New Taskmentioning

confidence: 99%

Response Selection for Multi-Party Conversations with Dynamic Topic Tracking

Wang¹,

Hoi

Joty

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

While participants in a multi-party multi-turn conversation simultaneously engage in multiple conversation topics, existing response selection methods are developed mainly focusing on a two-party single-conversation scenario. Hence, the prolongation and transition of conversation topics are ignored by current methods. In this work, we frame response selection as a dynamic topic tracking task to match the topic between the response and relevant conversation context. With this new formulation, we propose a novel multi-task learning framework that supports efficient encoding through large pretrained models with only two utterances at once to perform dynamic topic disentanglement and response selection. We also propose Topic-BERT an essential pretraining step to embed topic information into BERT with self-supervised learning. Experimental results on the DSTC-8 Ubuntu IRC dataset show state-of-the-art results in response selection and topic disentanglement tasks outperforming existing methods by a good margin. 1

show abstract

An Effective Domain Adaptive Post-Training Method for BERT in Response Selection

Cited by 72 publications

References 0 publications

Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models

Exploring the Data Efficiency of Cross-Lingual Post-Training in Pretrained Language Models

VD-BERT: A Unified Vision and Dialog Transformer with BERT

Response Selection for Multi-Party Conversations with Dynamic Topic Tracking

Contact Info

Product

Resources

About