Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snapshots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum—those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently “refreshed” as new data arrives, without the need for retraining from scratch.

This paper investigates how cognition facilitates the adoption of new words through a study of the large-scale Reddit corpus, which contains written, threaded conversations conducted over the internet. Parameters for the cognitive architecture are estimated. Using ACT-R's account of declarative memory, the activation of memory chunks representing words is traced and compared to usage statistics sampled from a year of data. Potential values for decay and retrieval threshold are identified according to model fit and growth rates of word adoption. The resulting estimate for the decay parameter, d, is 0.22, and the estimate for the retrieval threshold parameter, rt, lies between 3.4 and 4.5.

Why do bilinguals switch languages within a sentence? The present observational study asks whether word surprisal and word entropy predict code-switching in bilingual written conversation. We describe and model a new dataset of Chinese-English text with 1476 clean code-switched sentences, translated back into Chinese. The model includes known control variables together with word surprisal and word entropy. We found that word surprisal, but not entropy, is a significant predictor that explains code-switching above and beyond other well-known predictors. We also found sentence length to be a significant predictor, which has been related to sentence complexity. We propose high cognitive effort as a reason for code-switching, as it leaves fewer resources for inhibition of the alternative language. We also corroborate previous findings, but this time using a computational model of surprisal, a new language pair, and doing so for written language.

We examine working memory use and incrementality using acognitive model of grammatical encoding. Our model combinesan empirically validated framework, ACT-R, with a linguistictheory, Combinatory Categorial Grammar, to target thatphase of language production. By building the model with theSwitchboard corpus, it can attempt to realize a larger set ofsentences. With this methodology, different strategies may becompared according to the similarity of the model’s sentencesto the test sentences. In this way, the model can still be evaluatedby its fit to human data, without overfitting to individualexperiments. The results show that while having more workingmemory available improves performance, using less workingmemory during realization is correlated with a closer fit,even after controlling for sentence complexity. Further, sentencesrealized with a more incremental strategy are also moresimilar to the corpus sentences as measured by edit distance.As high incrementality is correlated with low working memoryusage, this study offers a possible mechanism by whichincrementality can be explained.

No abstract