Can you test me now? Equivalence of GMA tests on mobile and non‐mobile devices

Abstract

As technology continues to evolve, organizations seek to use personal electronics like smartphones for selection and assessment. While this promises to increase access to a more diverse applicant pool, research is needed to examine whether commonly used assessments function similarly on these devices as on a conventional computer. Contrary to past research, we did not find meaningful differences in general mental ability (GMA) test scores between device groups. We also observed few differences in item functioning between devices. Screen size had a positive, but marginal effect on test scores. These results are optimistic for the use of mobile devices in GMA testing, but additional research is needed to examine the functioning of alternative GMA tests administered on mobile devices.There is an increasing demand for the application of new technologies for the assessment of potential job applicants. As consumer adoption of computers and smartphones increases, companies seek to modify and develop assessments that make the best use of these technologies.According to a recent survey, roughly 81% of organizations reported using online assessments in 2014 (Kantrowitz, 2014). Although the current adoption of mobile assessments is less prevalent (Arthur, Doverspike, Munoz, Taylor, & Carr, 2014), 43% of HR professionals indicate that they would use assessments that were available for use on mobile devices (Kantrowitz, 2014). The demand among organizations for mobile testing has continued to increase, with one consulting firm estimating that use of mobile tests have increased by 44% in the customer service industry over the past 2 years (LaPort, Huynh, Stemer, Ryer, & Int J Select Assess. 2017;25:61-71 wileyonlinelibrary.com/journal/ijsa

Personnel selection in the digital age: a review of validity and applicant reactions, and future research challenges

“…They reported equivalence for both personality and cognitive tests, save for the observation of lower scores on cognitive tests completed on mobile devices. Brown and Grossenbacher (2017) also found measurement equivalence form completion on mobile and non-mobile devices, and also did not replicate the systematic score differentiation reported by Arthur et al (2014). King, Ryan, Kantrowitz, Grelle and Dainis (2015) in further contrast, did not show measurement equivalence for a cognitive test completed on a PC versus a mobile device.…”

Web-based and mixed-mode cognitive large-scale assessments in higher education: An evaluation of selection bias, measurement bias, and prediction bias

“…Moreover, various disturbances such as background noise or other people being able to see the test taker’s responses can potentially further influence the test-taking behavior (see Gnambs and Kaspar, 2015 , for respective evidence in the context of survey research). Finally, technological differences such as different screen sizes or input devices (e.g., mouse versus touchscreen) might introduce further construct-irrelevant variance that could distort measurements in unsupervised web-based settings, particularly when the assessment can be accessed using mobile and non-mobile devices (Brown and Grossenbacher, 2017 ). All this taken together limits the comparability of test scores.…”

The testing environment as an explanation for unproctored internet‐based testing device‐type effects

“…Despite these seemingly straightforward findings, Arthur et al. (2018) noted an oddity about this literature; specifically, whereas UIT device‐type effects have been consistently demonstrated by operational studies (e.g., Arthur et al., 2014; Wood et al., 2015), those in nonoperational settings such as research laboratories have consistently demonstrated no such effects (e.g., Arthur, et al., 2018; Brown & Grossenbacher, 2017; Parker & Meade, 2015). That is, for this latter group of studies, there were no differences in cognitive test scores between those who completed the test on ‘mobile’ versus ‘nonmobile’ devices.…”