A Systematic Study into the Factors that Affect the Predictive Accuracy of Multilevel VAR(1) Models

Abstract

The use of multilevel VAR(1) models to unravel within-individual process dynamics is gaining momentum in psychological research. These models accommodate the structure of intensive longitudinal datasets in which repeated measurements are nested within individuals. They estimate within-individual auto-and cross-regressive relationships while incorporating and using information about the distributions of these effects across individuals. An important quality feature of the obtained estimates pertains to how well they generalize to unseen data. Bulteel and colleagues (Psychol Methods 23(4):740-756, 2018a) showed that this feature can be assessed through a cross-validation approach, yielding a predictive accuracy measure. In this article, we follow up on their results, by performing three simulation studies that allow to systematically study five factors that likely affect the predictive accuracy of multilevel VAR(1) models: (i) the number of measurement occasions per person, (ii) the number of persons, (iii) the number of variables, (iv) the contemporaneous collinearity between the variables, and (v) the distributional shape of the individual differences in the VAR(1) parameters (i.e., normal versus multimodal distributions). Simulation results show that pooling information across individuals and using multilevel techniques prevent overfitting. Also, we show that when variables are expected to show strong contemporaneous correlations, performing multilevel VAR(1) in a reduced variable space can be useful. Furthermore, results reveal that multilevel VAR(1) models with random effects have a better predictive performance than person-specific VAR(1) models when the sample includes groups of individuals that share similar dynamics.

Is daily‐life stress reactivity a measure of stress recovery? An investigation of laboratory and daily‐life stress

“…To assess hypothesis 3, whether stress recovery in the laboratory was a better predictor than reactivity, we performed a blocked cross‐validation with R (Bergmeir & Benítez, 2012; Lafit et al., 2021; R Core Team, 2021; Roberts et al., 2017). We did this to compare the mean squared prediction error of the model with the laboratory reactivity (i.e.…”

“…The multilevel models were fitted on the training set, the prediction values were calculated using the test set. A mean square predictive error (MSPE) value was calculated as the average squared prediction error of the 10 blocks across all participants (see Lafit et al., 2021 for details). The MSPE is a measure of the ability of the training set in predicting the values of the outcome in the testing set, with a lower value reflecting a better predictive accuracy by the model.…”

“…blocked cross-validation with R (Bergmeir & Benítez, 2012;Lafit et al, 2021;R Core Team, 2021;Roberts et al, 2017). We did this to compare the mean squared prediction error of the model with the laboratory reactivity (i.e.…”

Guest Editors’ Introduction to the Special Issue on Forecasting with Intensive Longitudinal Data

“…While this general theme is certainly familiar, it leads to some interesting considerations in the context of forecasting and prediction as distinct from estimation and inference. Lafit et al (2021) nicely characterize this tension in the context of multilevel VAR. More generally, while there is a consensus that models for ILD should accommodate individual-specific parameterizations (e.g., via random effects), there are open questions as to the conditions under which prediction will be more accurate using individual-specific versus aggregate model properties (e.g., the predicted random effects or the fixed effects).…”

“…Chow et al (2022) discuss this problem from the perspective of control theory. Lafit et al (2021) provide insights about how both model and data characteristics affect prediction with multilevel VAR, while Nestler and Hamburg (2021) discuss some analytical results in the context of mixed-effects models. In many applications, accurate forecasts may lead to an appropriate course of action even if we lack a precise understanding of how those actions affect a system's dynamics (e.g., in the case of alcohol and substance use).…”