Who Stepped in It?

Abstract

Data are undeniably useful for answering many interesting and important questions, but data alone are not enough. Data without theory has been the source of a large (and growing) number of data miscues, missteps, and mishaps. We should resist the temptation to believe that data can answer all questions, and that more data means more reliable answers. Data can have errors and omissions or be irrelevant. In addition, patterns discovered in the past will vanish in the future unless there is an underlying reason for the pattern. Backtesting models in the stock market is particularly pernicious because it is so easy to find coincidental patterns that turn out to be expensive mistakes. This endemic problem has now spread far and wide because there are so much data that can be used by academic, business, and government researchers to discover phantom patterns.