Self-Selection as an Explanation for General Mental Ability Test Score Differences between Mobile and Non-Mobile Devices in Observational Studies

Abstract

Past studies have reported inconsistent results regarding the effect of mobile devices on cognitive ability test scores. We investigate selection bias as a potential explanation for cognitive ability test score differences between applicants using mobile or non-mobile devices. The likelihood of using a mobile device was predicted by educational attainment (R = .71) and O*NET codes (R = .84), both of which are also related to cognitive ability. Controlling for selection bias using propensity score weights reduced the standardized mean difference in test scores from d = 0.58 to d = 0.25 in a sample of 76,948 job applicants. The mobile device effect was further minimized when weighting using post-stratification (d = 0.10). This suggest that contradictory findings in past studies on mobile device effects are likely explained by selection bias in non-experimental studies. In practice, applicants with greater educational attainment were less likely to complete pre-hire assessments with a mobile device and tend to score higher on cognitive tests. Mobile use was also more common among applicants for lower complexity jobs which tend to attractapplicant pools with lower cognitive test scores on average. Therefore, it is important to control for demographic and occupational differences between mobile and non-mobile test takers when analyzing operational data. Propensity score weighting and post-stratification are useful for reducing the impact of selection bias in real-world, observational data. We also strongly recommend the use of random assignment in order to prevent selection bias in future research and test development