Hierachical Resampling for Bagging in Multi-Study Prediction with Applications to Human Neurochemical Sensing

Abstract

Prediction settings with multiple studies have become increasingly common. Ensembling models trained on individual studies has been shown to improve replicability in new studies. Motivated by a groundbreaking new technology in human neuroscience, we introduce two generalizations of multi-study ensemble predictions. First, while existing methods weight ensemble elements by cross-study prediction performance, we extend weighting schemes to also incorporate covariate similarity between training data and target validation studies. Second, we introduce a hierarchical resampling scheme to generate pseudo-study replicates ("study straps") and ensemble classifiers trained on these rather than the original studies themselves. We demonstrate analytically that existing methods are special cases. Through a tuning parameter, our approach forms a continuum between merging all training data and training with existing multistudy ensembles. Leveraging this continuum helps accommodate different levels of between-study heterogeneity.Our methods are motivated by the application of Voltammetry in humans. This technique records electrical brain measurements and converts signals into neurotransmitter concentration estimates using a prediction model. Using this model in practice presents a crossstudy challenge, for which we show marked improvements after application of our methods. We verify our methods in simulations and provide the studyStrap R package. * NSF-DMS1810829 † T32 AI 007358 ‡ NIH, R01 DA048096; NIH, R01 MH121099; NIH, R01 NS092701; NIH, 5KL2TR00142 § WFSOM, Phys/Pharm Neurosurgery

Optimal ensemble construction for multistudy prediction with applications to mortality estimation

“…10,13 The domain generalization literature focuses on leveraging multiple datasets in model training to improve model generalizability and enhance prediction performance on a new, unknown, but related, "domain". 14 Our methods are inspired both by this vast literature in transfer learning as well as related methods in "multistudy" statistics that draw upon multiple data sources in inference, 15,16 supervised prediction, 7,17,18 and unsupervised learning. 19 Previous work in this area proposed multistudy ensembling in conjunction with a generalization of stacking, an ensemble weight estimation method, 20 as a flexible strategy to aggregate information from different studies.…”

“…19 Previous work in this area proposed multistudy ensembling in conjunction with a generalization of stacking, an ensemble weight estimation method, 20 as a flexible strategy to aggregate information from different studies. 6,7,17,18 The approach involves two separate stages: (A) training one or more models on each study separately, and (B) constructing an ensemble prediction rule that is a weighted average of the predictions from each of the study-specific models. The ensemble weights are estimated in step B through "multistudy stacking," (MSS) by regressing the outcome of all training studies against the predictions of each of these models, using all available training data sets.…”