Prospect theory analysis of guessing in multiple choice tests

Abstract

The guessing of answers in multiple choice tests adds random error to the variance of the test scores, lowering their reliability. Formula scoring rules that penalize for wrong guesses are frequently used to solve this problem. This paper uses prospect theory to analyze scoring rules from a decision-making perspective and focuses on the effects of framing on the tendency to guess. In three experiments participants were presented with hypothetical test situations and were asked to indicate the degree of certainty that they thought was required for them to answer a question. In accordance with the framing hypothesis, participants tended to guess more when they anticipated a low grade and therefore considered themselves to be in the loss domain, or when the scoring rule caused the situation to be framed as entailing potential losses. The last experiment replicated these results with a task that resembles an actual test. Copyright # 2002 John Wiley & Sons, Ltd.key words multiple choice tests; framing effect; decision making; scoring rules Most standardized aptitude and achievement tests and many academic exams in schools and universities consist of multiple choice items. They are mainly used because their scoring is both easy and objective. Research on educational measurement has dealt extensively with scoring methods for these tests, aiming to increase their reliability and validity (Diamond and Evans, 1973;Frary, 1982; Rowley and Traub, 1977). We suggest that, in addition to the usual psychometric analyses, exams should also be considered as situations in which people face a series of decisions under uncertainty. Therefore the characteristics of behavioral decisionmaking processes that affect risk taking may shape people's behavior during tests. Individual differences in examinees' tendencies to take risks and the use of different scoring methods are likely to alter the patterns of responses systematically. This can provide advantages or disadvantages for certain groups of examinees, which may lower the tests' validity. The series of experiments reported here demonstrates that examinees' expectations about the outcome of a test and the scoring methods that are employed in an exam indeed create framing effects that affect the tendency to take risks and to guess in exams.

Optimal correction for guessing in multiple-choice tests

“…For instance, it could easily adopt the extended IRT model of San Martín, del Pino, and De Boeck (2006) where the guessing parameter (or probability of correct guess) does not only depend on the item, but also on the ability of individuals. Another interesting extension would follow Bereby-Meyer et al (2002), relaxing the assumption of expected utility maximization by assuming that the utility of rewards is smaller than the disutility of penalties.…”

“…There exist other possible explanations for this evidence: the influence of directions given before the exam (e.g., Budescu & Bar-Hillel, 1993), framing of the scoring rule (e.g., Bereby-Meyer, Meyer, & Flascher, 2002) and personality (e.g., Avila & Torrubia, 2004). However, in this paper we will develop an explanation based on the idea that under formula scoring rational examinees may try to avoid the risk of being penalized by omitting some items in which they have partial knowledge.…”

Guessing and risk attitude in L2 vocabulary tests

“…Anderson and Freebody (1981: 109-110) believe that "the notable problem with the yes/no [vocabulary] task is that scores of individuals will be influenced markedly by differences in tendency to take risks in the face of uncertainty". Several other authors also simply postulate a strong positive relation between risk taking and guessing (Bachman & Palmer, 1996;Espinosa & Gardeazabal, 2007) although there is some evidence for this assumption from research with multiplechoice tests (Alnabhan, 2002;Bereby-Meyer, Meyer & Falscher, 2002;Koehler, 1974;Slakter, Crehan & Koehler, 1975;Walker & Thompson, 2001;Whitmore, Shore & Smith, 2004).…”

“…When they perceive themselves in a domain of loss, they will be risk taking. Bereby-Meyer et al (2002) were the first to test the application of prospect theory to guessing and risk behaviour in discrete tests empirically. They operationalized gain and loss domains in student expectations of high and low grades, respectively.…”

Optimal Weighting for Exam Composition

“…Conventionally, when multiple choice tests evaluation was introduced , the scoring was done using a traditional number right (NR) scoring method [17]. Where only the correct responses are scored positively and considered in the test scoring neglecting the incorrect answers and unanswered ones totally.…”