Role of test motivation in intelligence testing

Abstract

Intelligence tests are widely assumed to measure maximal intellectual performance, and predictive associations between intelligence quotient (IQ) scores and later-life outcomes are typically interpreted as unbiased estimates of the effect of intellectual ability on academic, professional, and social life outcomes. The current investigation critically examines these assumptions and finds evidence against both. First, we examined whether motivation is less than maximal on intelligence tests administered in the context of low-stakes research situations. Specifically, we completed a metaanalysis of random-assignment experiments testing the effects of material incentives on intelligence-test performance on a collective 2,008 participants. Incentives increased IQ scores by an average of 0.64 SD, with larger effects for individuals with lower baseline IQ scores. Second, we tested whether individual differences in motivation during IQ testing can spuriously inflate the predictive validity of intelligence for life outcomes. Trained observers rated test motivation among 251 adolescent boys completing intelligence tests using a 15-min "thin-slice" video sample. IQ score predicted life outcomes, including academic performance in adolescence and criminal convictions, employment, and years of education in early adulthood. After adjusting for the influence of test motivation, however, the predictive validity of intelligence for life outcomes was significantly diminished, particularly for nonacademic outcomes. Collectively, our findings suggest that, under low-stakes research conditions, some individuals try harder than others, and, in this context, test motivation can act as a third-variable confound that inflates estimates of the predictive validity of intelligence for life outcomes.O ne of the most robust social science findings of the 20th century is that intelligence quotient (IQ) scores predict a broad range of life outcomes, including academic performance, years of education, physical health and longevity, and job performance (1-7). The predictive power of IQ for such diverse outcomes suggests intelligence as a parsimonious explanation for individual and group differences in overall competence.However, what is intelligence? Boring's now famous reply to this question was that "intelligence as a measurable capacity must at the start be defined as the capacity to do well in an intelligence test. Intelligence is what the tests test." (ref. 8, p. 35). This early comment augured the now widespread conflation of the terms "IQ" and "intelligence," an unfortunate confusion we aim to illuminate in the current investigation.Intelligence has more recently-and more usefully-been defined as the "ability to understand complex ideas, to adapt effectively to the environment, to learn from experience, to engage in various forms of reasoning, to overcome obstacles by taking thought" (ref. 5, p. 77). IQ scores, in contrast, measure the performance of individuals on tests designed to assess intelligence. That is, IQ is an observed, manifest ...

Does IQ = IQ? Comparability of Intelligence Test Scores in Typically Developing Children

“…In our study, however, intelligence test score differences were not associated with the time interval between test sessions or the order of test administration. To further examine possible interaction effects between examinees and temporal aspects of the testing session, future studies might additionally assess motivation during the testing situation (Duckworth, Quinn, Lynam, Loeber, & Stouthamer-Loeber, 2011), the time of day (Gupta, 1991), as well as "morningness" and "eveningness" (Goldstein, Hahn, Hasher, Wiprzycka, & Zelazo, 2007).…”

“…In our study, however, intelligence test score differences were not associated with the time interval between test sessions or the order of test administration. To further examine possible interaction effects between examinees and temporal aspects of the testing session, future studies might additionally assess motivation during the testing situation (Duckworth, Quinn, Lynam, Loeber, & Stouthamer-Loeber, 2011), the time of day (Gupta, 1991), as well as "morningness" and "eveningness" (Goldstein, Hahn, Hasher, Wiprzycka, & Zelazo, 2007).Further, interaction effects between the examinee's ability level and intelligence test score characteristics may be present, such as floor effects for children with intellectual disabilities or ceiling effects for gifted children. In our study we analyzed the comparability of intelligence test scores in three qualitative nominal intelligence levels (i.e., below average, average, above average).…”

Non-cognitive skills and individual earnings: new evidence from PIAAC

“…Indeed, non-cognitive skills have been found to be important for performance on IQ tests due to the low stakes nature of the latter. As shown by Borghans et al (2008); Duckworth et al (2011);Gneezy and Rustichini (2000) and Segal (2012), scores of low performers on IQ tests can be improved by offering a reward. As mentioned above, some of this literature have explicitly proposed a decomposition of test scores into a cognitive and a non-cognitive component, see Borghans and Schils (2012).…”

“…The first one is, how should one interpret scores of achievement tests? According to the conclusions of Borghans et al (2011), Duckworth et al (2011) and Segal (2012), scores on low-stakes achievement tests should not be interpreted as a measure of cognitive skills but as a broader notion of human capital that also includes noncognitive skills. The second question concerns the possibility of obtaining separate measures for each type of skill.…”

“…However, recent research has provided alternative measures of non-cognitive skills. First, Borghans et al (2011), Duckworth et al (2011) and Segal (2012) show that scores on low-stake achievement tests are not only driven by high cognitive skills but also by high non-cognitive skills. This observation provides an opportunity to have more detailed measures of human capital that specifically distinguish between cognitive and non-cognitive skills.…”