Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media

Abstract

Language use differs between domains and even within a domain, language use changes over time. For pre-trained language models like BERT, domain adaptation through continued pre-training has been shown to improve performance on in-domain downstream tasks. In this article, we investigate whether temporal adaptation can bring additional benefits. For this purpose, we introduce a corpus of social media comments sampled over three years. It contains unlabelled data for adaptation and evaluation on an upstream masked language modelling task as well as labelled data for finetuning and evaluation on a downstream document classification task. We find that temporality matters for both tasks: temporal adaptation improves upstream and temporal fine-tuning downstream task performance. Time-specific models generally perform better on past than on future test sets, which matches evidence on the bursty usage of topical words. However, adapting BERT to time and domain does not improve performance on the downstream task over only adapting to domain. Tokenlevel analysis shows that temporal adaptation captures event-driven changes in language use in the downstream task, but not those changes that are actually relevant to task performance. Based on our findings, we discuss when temporal adaptation may be more effective.

Temporal Attention for Language Models

“…Several recent studies have explored and evaluated the generalization ability of language models to time (Röttger and Pierrehumbert, 2021;Lazaridou et al, 2021;Agarwal and Nenkova, 2021). To better handle continuously evolving web content, Hombaiah et al ( 2021) performed incremental training.…”

“…They focused on two temporal tasks: semantic change detection and sentence time prediction. Others focused on document classification by using wordlevel temporal embeddings (Huang and Paul, 2019) and adapting pretrained BERT models to domain and time (Röttger and Pierrehumbert, 2021).…”

“…The "static" nature of existing LMs makes them unaware of time, and in particular unware of language changes that occur over time. This prevents such models from adapting to time and generalizing temporally (Röttger and Pierrehumbert, 2021;Lazaridou et al, 2021;Hombaiah et al, 2021;Dhingra et al, 2021;Agarwal and Nenkova, 2021), abilities that were shown to be important for many tasks in NLP and Information Retrieval (Kanhabua and Anand, 2016;Rosin et al, 2017;Huang and Paul, 2019;Röttger and Pierrehumbert, 2021;Savov et al, 2021). Recently, to create time-aware models, the NLP community has started to use time as a feature in training and fine-tuning language models (Dhingra et al, 2021;Rosin et al, 2021).…”

Temporal Effects on Pre-trained Models for Language Processing Tasks

“…The global changes captured with language model perplexity and analysis of individual words cannot indicate how these changes impact the performance of a model trained for a given task. Röttger and Pierrehumbert (2021) present a meticulously executed study of how domain change (topic of discussion) influences both language models and a downstream classification task. They show that even big changes in language model perplexity may lead to small changes in downstream task performance.…”

“…The only problem is that there is a single test year chosen and any anomaly in that test year may lead to misleading results. An evaluation setup much like the one in Huang and Paul (2018) or the much more recent work in Röttger and Pierrehumbert (2021) is needed to draw robust conclusions. As noted above, we adopt their setup with some changes and additionally provide visualization plots to easily interpret trends.…”