Exploration, Inference, and Prediction in Neuroscience and Biomedicine

Abstract

The last decades saw dramatic progress in brain research. These advances were often buttressed by probing single variables to make circumscribed discoveries, typically through null hypothesis significance testing. New ways for generating massive data fueled tension between the traditional methodology, used to infer statistically relevant effects in carefully-chosen variables, and pattern-learning algorithms, used to identify predictive signatures by searching through abundant information. In this article, we detail the antagonistic philosophies behind two quantitative approaches: certifying robust effects in understandable variables, and evaluating how accurately a built model can forecast future outcomes. We discourage choosing analysis tools via categories like 'statistics' or 'machine learning'. Rather, to establish reproducible knowledge about the brain, we advocate prioritizing tools in view of the core motivation of each quantitative analysis: aiming towards mechanistic insight, or optimizing predictive accuracy. The emergence of richer datasets alters everyday data-analysis practicesThere is a burgeoning controversy in neuroscience on what data analysis should be about.Similar to many other biomedical disciplines, there are differing perspectives among researchers, clinicians, and regulators about the best approaches to make sense of these unprecedented data resources. Traditional statistical approaches, such as null hypothesis significance testing, were introduced in a time of data scarcity and have been revisited, revised, or even urged to be abandoned. Currently, a growing literature advertises predictive pattern-learning algorithms hailed to provide some traction on the data deluge [2,3]. Such modeling tools for prediction are increasingly discussed in particular fields of neuroscience [for some excellent sources see: 4, 5-9].Ensuing friction is aggravated by the incongruent historical trajectories of mainstream statistics and emerging pattern-learning algorithms -the former long centered on significance testing procedures to obtain p-values, the latter with a stronger heritage in computer science [10][11][12]. We argue here that the endeavor of sorting each analysis tool into categories like 'statistics' or 'machine-learning' is futile [13,14].Take for instance ordinary linear regression, as it is routinely applied by many neuroscientists. The same tool and its underlying mathematical prosthetics can be used to achieve three diverging goals [15, pp. 82-83, 16, ch. 4.12]: a) exploration, to get a first broad impression of the dependencies between a set of measured variables in the data at hand, b) inference, to discern which particular input variables contribute to the target variable beyond chance level, and c) prediction, to enable statements about how well target variables can be guessed based on data measured in the future.Confusion can arise because it is the motivation for using linear regression that differs between these scenarios. The mathematical mechanics underlying model parameter fitting a...

Predictive regression modeling with MEG/EEG: from source power to signals and cognitive states

“…This raises the question if the specificity of a Riemannian model could be enhanced in a similar way. Ultimately, what model to prefer, therefore, clearly depends on the strategic goal of the analysis (Bzdok et al, 2018;Bzdok and Ioannidis, 2019) and cannot be globally decided.…”

Automated assessment of psychiatric disorders using speech: A systematic review

“…Null-hypothesis models isolate variables deemed important above a relatively arbitrary P value, and can be incongruent with the variables that maximize predictions in new settings. 49 Significant differences are usually considered more useful for scientific inference than prediction. Predictive studies train models on a subset of the data and test performance on the rest of the data not used for training and therefore give insight into how well they may generalize to new individuals.…”

“…Both types of models are built using automatically extracted acoustic features. Null‐hypothesis models isolate variables deemed important above a relatively arbitrary P value, and can be incongruent with the variables that maximize predictions in new settings . Significant differences are usually considered more useful for scientific inference than prediction.…”

Your evidence? Machine learning algorithms for medical diagnosis and prediction

“…Yet, there is evidently a broad spectrum of conceptually and in particular technically different methods that are currently being used in medical applications including imaging neuroscience. For a more detailed overview, we would like to point the reader to other, more specialized publications (Bishop, ; Bzdok & Ioannidis, ; Choudhary & Gianey, ; James, Witten, Hastie, & Tibshirani, ; Jordan & Mitchell, ). What is critical to note in the current context, though, is that there is generally an inverse relation between the potential accuracy or performance of machine‐learning algorithms on one hand and their interpretability on the other.…”