Are Internet and Paper-and-Pencil Personality Tests Truly Comparable?

Abstract

Recently, the use of technology in assessment for personnel selection has increased dramatically. An important consideration is whether test scores obtained via Internet administration are psychometrically equivalent to those obtained by the more traditional paper-and-pencil format. Our results suggest that there is comparability of scores for many personality constructs, including conscientiousness. However, invariance was not found for some scales between persons allowed to choose formats and those not given a choice of formats. As testing-format preference may be related to membership in federally protected demographic groups, this latter finding was somewhat troubling. Additionally, we illustrate the use of an experimental laboratory design to investigate possible causes of a lack of measurement invariance in Internet and paper-and-pencil comparisons.

The equivalence of Internet versus paper-based surveys in IT/IS adoption research in collectivistic cultures: the impact of satisficing

“…Miles 1995, Leidner andKayworth 2006), while other studies indicate that survey modes indeed can affect the results (e.g. Fouladi et al 2002, Buchanan et al 2005, Meade et al 2007. Although the results remain inconclusive, these studies have contributed to our understanding of MI between paper and online surveys.…”

“…King and Miles 1995, Booth--Kewley et al 2007, Meade et al 2007), but most of this research was conducted within individualistic cultures. Few empirical studies have investigated whether paper and online surveys produced convergent results in collectivistic cultures.…”

Measurement Invariance Study of the Training Satisfaction Questionnaire (TSQ)

“…a lack of explicit measurement models of satisfaction scales that are directly related to empirical definitions of the dimensions included in the theoretical models, and the consequences of considering the effects of categorizing supposedly continuous variables), we present, based on a second-order factor satisfaction measurement model (Holgado, Chacón, Barbero, & Sanduvete, 2006), an empirical measurement invariance study of participants' satisfaction in different organizations and in different training methods (Meade, Michels, & Lautenschlager, 2007;Vanderberg & Lance, 2000). Our approach takes into account both the factor structure of the measured construct and the ordinal nature of the recorded data.…”

Towards a Comprehensive Procedure for Developing Measurement Scales for Cross-Cultural Management Research

“…Vandenberg and Lance (2000) suggest that equivalence tests should be performed when comparing different "populations" or "groups", whereas other authors refer to "cultural groups", "cultural samples" (e.g., Schaffer & Riordan, 2003;Vandenberg, 2002), "cross-national" groups (Steenkamp & Baumgartner, 1998) or "ethnic groups within countries" (Van de Vijver, 1998). As suggested by Meade and colleagues (Meade, Michels, & Lautenschlager, 2007), tests of measurement invariance (the generic term for such equivalence in the research tradition that uses structural equation modeling for these tests) have often proceeded in an atheoretical fashion in that between-group comparisons are made without a priori notions of whether equivalence would exist.…”