Using synthetic data to improve the reproducibility of statistical results in psychological research.

Abstract

In recent years, psychological research has faced a credibility crisis, and open data are often regarded as an important step toward a more reproducible psychological science. However, privacy concerns are among the main reasons that prevent data sharing. Synthetic data procedures, which are based on the multiple imputation (MI) approach to missing data, can be used to replace sensitive data with simulated values, which can be analyzed in place of the original data. One crucial requirement of this approach is that the synthesis model is correctly specified. In this article, we investigated the statistical properties of synthetic data with a particular emphasis on the reproducibility of statistical results. To this end, we compared conventional approaches to synthetic data based on MI with a data-augmented approach (DA-MI) that attempts to combine the advantages of masking methods and synthetic data, thus making the procedure more robust to misspecification. In multiple simulation studies, we found that the good properties of the MI approach strongly depend on the correct specification of the synthesis model, whereas the DA-MI approach can provide useful results even under various types of misspecification. This suggests that the DA-MI approach to synthetic data can provide an important tool that can be used to facilitate data sharing and improve reproducibility in psychological research. In a working example, we also demonstrate the implementation of these approaches in widely available software, and we provide recommendations for practice.

The criterion‐related validity of conscientiousness in personnel selection: A meta‐analytic reality check

“…Sixth, such publications should adhere to open sciences practices. Sophisticated methods of de‐identification and synthetic data procedures have been proposed that address legitimate privacy concerns while ensuring the ability to verify results (Grund et al, 2022; Walsh et al, 2018). If more studies meeting these requirements were available, a more dependable verdict concerning the predictive validity of self‐report measures of personality and its moderators were possible.…”

“…that address legitimate privacy concerns while ensuring the ability to verify results (Grund et al, 2022;Walsh et al, 2018). If more studies meeting these requirements were available, a more dependable verdict concerning the predictive validity of self-report measures of personality and its moderators were possible.…”

Comparing Different Trend Estimation Approaches in International Large-Scale Assessment Studies

“…The replication code for the empirical application can be found in Supplement S2 at https://osf.io/n5zm6/?view_only=086ea651bbea49bb8b2aae44e3971db8. As the original data can only be downloaded by individual researchers from the OECD websites and we are not allowed to make our processed PISA data publicly available, we created synthetic datasets (see Grund et al, 2022) that are very similar to the original datasets. The generation of the synthetic datasets is described in Appendix A.…”

“…To enable the replicability of the results of the empirical example by independent researchers, we created synthetic Datasets for the two PISA assessments that strongly resemble the original data. The principle of data generation relied on the approach of Jiang et al (2022), which was also investigated by Grund et al (2022). Synthetic datasets were produced at the country level for each of the three cognitive domains and the two PISA assessments, PISA 2006 andPISA 2009. To also represent the balanced incomplete block design in the synthetic datasets, we applied the synthesis model at the level of each administered booklet in the test.…”

“…Synthetic data Xsyn can be formally derived as a simulation draw from the posterior distribution (Grund et al, 2022) 𝐗 syn ∼ P(𝐗|𝐙) = P(𝐗, 𝐙) P(𝐙) (A1) Jiang et al (2022) proposed the brilliant idea of including noisy realizations of X and also using them as fixed variables. That is, they define the data-augmented variables 𝐗 * by adding random noise e to the original data…”