Establishing the Measurement Equivalence of Online Selection Assessments Delivered on Mobile versus Nonmobile Devices

Abstract

Recent usage data suggest job applicants are completing online selection assessments using mobile devices (e.g., smartphones) in greater numbers. To determine the appropriateness of this new technology, this study examined the measurement equivalence of selection assessments delivered on mobile and nonmobile devices (e.g., personal computers). Measurement invariance tests conducted with multigroup confirmatory factor analysis suggest mobile versions of a cognitive ability‐type assessment, two biodata assessments, a multimedia work simulation, and a text‐based situational judgment test appear to be equivalent to nonmobile versions. However, mobile device user latent means were half a standard deviation lower than their nonmobile counterparts for the situational judgment test. Implications for mobile device usage within selection and assessment are discussed.

The comparison of mobile devices to computers for web-based assessments

“…This is an important difference since prior research has shown that the user interface can have an impact on computer and web-based test scores (Bridgeman, Lennon, & Jackenthal, 2003;Huff, Cline, & Guynes, 2012;Květon, Jelínek, Vobořil, & Klimusová, 2007;Schroeders and Wilhelm, 2010;Wästlund et al's, 2008). More specifically to this research, very recent research have found differences in mean test scores between mobile devices and traditional computers (Arthur et al, 2014;Illingworth et al, 2014;Morelli et al, 2014;Sanchez & Branaghan, 2011;Sanchez & Goolsbee, 2010). However, the effect sizes of those differences have traditionally been small and very little is known as to why the differences exist at all.…”

“…What research that does exists (Arthur, Doverspike, Muñoz, Taylor, & Carr, 2014;Illingworth, Morelli, Scott, & Boyd, 2014;Morelli, Mahan, & Illingworth, 2014;Sanchez & Branaghan, 2011;Sanchez & Goolsbee, 2010) demonstrates that assessments can be impacted by the type of device that was used to completed it and that this impact is not necessarily consistent across different types of assessments. What little we know about the differences that exist is limited by the lack of experiments investigating issues that may cause differences on assessments across devices.…”

“…Research by Arthur et al (2014) revealed that 1.9% (almost 70,000 individuals) of a sample of more than 3.5 million completed unproctored assessments on a mobile device. Similarly, research by Illingworth et al (2014) revealed that just over .8%, or about 8000 applicants of their sample of over 900,000 completed pre-employment assessments on a mobile device whereas Morelli et al (2014) reported that .3% and .5% of their samples of over 209,000 and 375,000, respectively. If we assume similar trends in all other web-based assessments, such as surveys and web-based research studies, then a sizeable portion of the respondents are likely to attempt to complete the web-based assessments with mobile devices.…”

“…One of the earliest and most often explanations offered as to why differences between testing mediums exist has to do with scrolling (Bridgeman et al, 2003;Leeson, 2006;Morelli et al, 2014;Sanchez & Branaghan, 2011;Sanchez & Goolsbee, 2010;Wästlund et al, 2008). Basically, test takers who had to scroll in order to complete the test generally performed worse when compared to test takers who did not have to scroll to complete the assessment.…”

Felső tagozatos diákok papír-ceruza és számítógép alapú teszteredményeinek összehasonlító vizsgálata szövegértés, induktív gondolkodás és problémamegoldás terén

“…Amennyiben a pedagógiai adatfelvétel a számítógépek valamelyik fajtáján történik (pl. Schröders és Wilhelm, 2010;Morelli, Mahan és Illingworth, 2014), számítógépes mérés-értékelés (computerised, computer-based testing/assessment) valósul meg. A számítógépes tesztelés során a teljes teszt vagy a megoldandó item(ek) a számítógép képernyőjén jelenik/jelennek meg és a diák billentyűzet vagy egér használatával adja meg válaszát (Csapó és mtsai, 2008).…”