Time and the Value of Data

Abstract

Managers often believe that collecting more data will continually improve the accuracy of their machine learning models. However, we argue in this paper that when data lose relevance over time, it may be optimal to collect a limited amount of recent data instead of keeping around an infinite supply of older (less relevant) data. In addition, we argue that increasing the stock of data by including older datasets may, in fact, damage the model's accuracy. Expectedly, the model's accuracy improves by increasing the flow of data (defined as data collection rate); however, it requires other tradeoffs in terms of refreshing or retraining machine learning models more frequently.Using these results, we investigate how the business value created by machine learning models scales with data and when the stock of data establishes a sustainable competitive advantage. We argue that data's time-dependency weakens the barrier to entry that the stock of data creates. As a result, a competing firm equipped with a limited (yet sufficient) amount of recent data can develop more accurate models. This result, coupled with the fact that older datasets may deteriorate models' accuracy, suggests that created business value doesn't scale with the stock of available data unless the firm offloads less relevant data from its data repository. Consequently, a firm's growth policy should incorporate a balance between the stock of historical data and the flow of new data.We complement our theoretical results with an experiment. In the experiment, we use the simple yet widely used machine learning task known as next work prediction. We empirically measure the loss in the accuracy of a next word prediction model trained on datasets from various time periods. Our empirical measurements confirm the economic significance of the value decline over time. For example, 100MB of text data, after seven years, becomes as valuable as 50MB of current data for the next word prediction task.

The Editor and the Algorithm: Recommendation Technology in Online News

“…Moreover, our large-scale field experiment provides causal estimates by going beyond offline evaluations, which often form the basis for a majority of applied technical research (e.g., the studies reviewed in Jannach andJugovac 2019 andWu et al 2022). Such offline evaluations often suffer from endogeneity concerns because of the inability of historical data to account for fast-paced dynamics and customer feedback loops (Valavi et al 2022). More recently, this literature has run into severe methodological issues and problems of replication (Kapoor and Narayanan 2023), highlighting the need for credible field experiments.…”

Platform Competition and Information Sharing

“…It is not uncommon to consider a S-shaped relationship between data and the value it creates (Hagiu & Wright, 2020a;Parker et al, 2022;Posner & Weyl, 2018;Valavi, Hestness, Ardalani, & Iansiti, 2022). Consider, for example, the data-network value relationship depicted in Figure 3.…”