The Equivalence of Scores From Automated and Conventional Educational and Psychological Tests

Abstract

A literature review was conducted to determine the current state of knowledge concerning the effects of the computer administration of standardized educational and psychological tests on the psychometric properties of these instruments. Studies were grouped according to a number of factors relevant to the administration of tests by computer. Based on the studies reviewed, we arrived at the following conclusions: The rate at which test‐takers omit items in an automated test may differ from the rate at which they omit items in a conventional presentation. Scores on tests from automated versions of personality inventories such as the Minnesota Multiphasic Personality Inventory are lower than scores obtained in the conventional testing format. These differences may result in part from differing omit rates, as described above, but some of the differences may be caused by other factors. Scores from automated versions of speed tests are not likely to be comparable with scores from paper‐and‐pencil versions. The presentation of graphics in an automated test may have an effect on score equivalence. Such effects were obtained in studies using the Hidden Figures Test. However, in studies with three Armed Services Vocational Aptitude Battery (ASVAB) tests, effects were not found. Tests containing items based on reading passages can become more difficult when presented on a CRT. This was demonstrated in a single study with the ASVAB tests. The possibility of such asymmetric practice effects may make it wise to avoid conducting equating studies based on single‐group counterbalanced designs.

Psychometric properties and correlates of three computer aversion scales

“…First, computer-anxious clients who complete psychological tests administered by computer may produce test scores different from those produced by paper-and-pencil administration (see Hofer & Green, 1985;Honaker, 1987). There is evidence that persons who take tests by computer may alter their rate of omitting items (Mazzeo & Harvey, 1988), increase their "faking good" responses (Davis & Cowles, 1989), and elevate scores on negative affect scales such as those that assess depression and anxiety (George, Lankford, & Wilson, 1990). Second, acceptance of computer-based procedures-from expert systems to CAl and training applications-may also partially depend on mental health professionals' comfort with computer use (Meier, 1985).…”

A Historical Survey of Research Regarding Constructed-Response Formats

“…These differences could introduce construct-irrelevant variance due to the testing medium. The investigation of that question has received significant attention (Gallagher et al 2002;Horkay et al 2006;Mazzeo and Harvey 1988;Powers et al 1994;Puhan et al 2007;Wolfe et al 1993). …”

A Proposal on the Validation Model of Equivalence between PBLT and CBLT

“…Since computer is the medium of CBLT which can be presented interactively through many channels, the interaction and multimedia effects may also affect test performance and can be regarded as a potential element of CBLT validity. The study conducted by Mazzeo and Harvey (1988) found out that the number of screens, screen size, font and letter size, image resolution, and other complex ways of presentation could have significant impact on CBLT performance. McKee and Levinson (1990) pointed out that those computer-related modes of item presentation probably had greatly changed the nature of items and that a PBLT and the relevant CBLT probably were not testing the same thing.…”