Using synthetic data to improve the reproducibility of statistical results in psychological research

Abstract

In recent years, psychological research has faced a credibility crisis, and open data are often regarded as an important step toward a more reproducible psychological science. However, privacy concerns are among the main reasons that prevent data sharing. Synthetic data procedures, which are based on the multiple imputation (MI) approach to missing data, can be used to replace sensitive data with simulated values, which can be analyzed in place of the original data. One crucial requirement of this approach is that the synthesis model is correctly specified. In this article, we investigated the statistical properties of synthetic data with a particular emphasis on the reproducibility of statistical results. To this end, we compared conventional approaches to synthetic data based on MI with a data-augmented approach (DA-MI) that attempts to combine the advantages of masking methods and synthetic data, thus making the procedure more robust to misspecification. In multiple simulation studies, we found that the good properties of the MI approach strongly depend on the correct specification of the synthesis model, whereas the DA-MI approach can provide useful results even under various types of misspecification. This suggests that the DA-MI approach to synthetic data can provide an important tool that can be used to facilitate data sharing and improve reproducibility in psychological research. In a worked example, we also demonstrate the implementation of these approaches in widely available software, and we provide recommendations for practice.

Using synthetic data to improve the reproducibility of statistical results in psychological research.

“…To this end, we investigated the behavior of these methods both with correctly specified synthesis models and under different types of misspecification, reflecting applications of synthetic data in which the analyses were either consistent with the relations that were featured in the original analysis and specified in the synthesis model (reproducibility Stages 1 and 2) or went beyond them (Stages 3 and 4). The materials used to conduct this study, including the computer code and all syntax files, are provided on the OSF (https://osf.io/3a5uq/; Grund et al, 2021).…”

Accuracy and Precision of Fixed and Random Effects in Meta-Analyses of Randomized Control Trials for Continuous Outcomes

“…Although combining the raw data of multiple studies in individual participant meta-analyses is preferable from a methodological point of view (Riley et al, 2010), most psychological studies do not provide the respective raw data (Hardwicke et al, 2021;Nutu et al, 2019). Particularly, in clinical research often legal restrictions or ethical considerations prevent sharing the raw data (see Grund et al, 2021, for a potential remedy). Therefore, meta-analyses of summary statistics are the only viable solution in many situations.…”

Anonymiced Shareable Data: Using mice to Create and Analyze Multiply Imputed Synthetic Datasets

“…Generally, there are three ways to impute data: joint modeling, sequential modeling, and fully conditional specification [12,16,17]. With joint modeling, the entire joint distribution of the data is specified as a single multivariate distribution, and the imputations are drawn from this model.…”

“…Choosing an adequate imputation model to impute the data is paramount, as a flawed imputation model may drastically impact the validity of inferences [12,13]. Imputation models should be as flexible as possible to capture most of the patterns in the data, and to model possibly unanticipated data characteristics [16,24].…”

“…However, there is no such thing as a free lunch: data collectors have to put effort in creating highquality synthetic data. The quality of the synthetic data is highly dependent on the imputation models, and using flawed imputation models might bias subsequent analyses [11][12][13]. Conversely, if the models used to synthesize the data are able to preserve the relationships between the variables as in the original data, the synthetic data can be nearly as informative as the observed data.…”