Replicability of Machine Learning Models in the Social Sciences

Abstract

Abstract. Machine learning tools are increasingly used in social sciences and policy fields due to their increase in predictive accuracy. However, little research has been done on how well the models of machine learning methods replicate across samples. We compare machine learning methods with regression on the replicability of variable selection, along with predictive accuracy, using an empirical dataset as well as simulated data with additive, interaction, and non-linear squared terms added as predictors. Methods analyzed include support vector machines (SVM), random forests (RF), multivariate adaptive regression splines (MARS), and the regularized regression variants, least absolute shrinkage and selection operator (LASSO), and elastic net. In simulations with additive and linear interactions, machine learning methods performed similarly to regression in replicating predictors; they also performed mostly equal or below regression on measures of predictive accuracy. In simulations with square terms, machine learning methods SVM, RF, and MARS improved predictive accuracy and replicated predictors better than regression. Thus, in simulated datasets, the gap between machine learning methods and regression on predictive measures foreshadowed the gap in variable selection. In replications on the empirical dataset, however, improved prediction by machine learning methods was not accompanied by a visible improvement in replicability in variable selection. This disparity is explained by the overall explanatory power of the models. When predictors have small effects and noise predominates, improved global measures of prediction in a sample by machine learning methods may not lead to the robust selection of predictors; thus, in the presence of weak predictors and noise, regression remains a useful tool for model building and replication.

Predicting Students’ Academic Performance: A Comparison between Traditional MLR and Machine Learning Methods with PISA 2015

“…ML methods are a group of varied modelling techniques intended to help a system learn the patterns of the data, either with some or no theoretical assumptions about the population of study (Vijayakumar & Cheung, 2018). These methods can be loosely categorised into regression-based methods which produce more interpretable models (e.g.…”

“…support vector machine, or neural network models). Although methods in these two categories vary considerably, all ML algorithms, in general, prioritise predictive accuracy, which entails some similarities in characteristics that set them apart from traditional statistical methods (Vijayakumar & Cheung, 2018).…”

“…In traditional OLS models, there is a focus on producing an unbiased population estimate, which results in having a fairly large variance, and compromises predictive accuracy. However, ML methods "choose their parameters to minimise total predictive error and so can trade off an increase in bias for a decrease in variance" (Vijayakumar & Cheung, 2018), increasing predictive accuracy.…”

“…Using a large data set from the German GESIS Panel, Vijayakumar and Cheung (2018) compared prediction performance of several ML methods and multiple regression for predicting individuals' well-being, and found that ML methods only performed better than MLR when the underlying structures were complex with non-linear patterns. Thus, evidence points to sample size and number of predictors used as important considerations when comparing between ML methods and conventional methods.…”

From Abstract Symbols to Emotional (In-)Sights: An Eye Tracking Study on the Effects of Emotional Vignettes and Pictures

“…In this context, machine learning assisted methods of predictive modeling offer a promising and valuable alternative or complementary perspective (e.g., Yarkoni and Westfall, 2017;Vijayakumar and Cheung, 2018). Since the approach allows working with many intercorrelated variables and non-linear data patterns (Goodman et al, 2016;Cheung and Jak, 2018), it is particularly well suited for analyzing effects of literary texts on reading behavior (Jacobs, 2018a,b;Xue et al, 2019).…”