A Complexity Analysis of Items From a Survey of Academic Achievement in the Life Sciences.

Abstract

The difficulty of 44 items from the life sciences subscale of the NAEP 1985–86 science assessment was analyzed in terms of item attributes and science educators' judgments of difficulty. The attributes included ratings of various characteristics of the items' text and option set, the items' cognitive demand, and the level of knowledge required by items. Science educators' mean judgment of item difficulty, which accounted for 52% of the variance, was the best single predictor of item difficulty. Combining item attribute information with educators' judgments of item difficulty improved the prediction of item difficulty on the order of 7% to 15% of the variance. When item difficulty was modeled in terms of discrete item attributes (global judgments of item difficulty not included in the model), the level of knowledge required was an important determinant of difficulty, while cognitive demand was not. The implications of these results for construct validation and for test design are discussed.

Examining alignment : national and local assessments and the common core state standards in mathematics.

“…In an effort to determine whether the CCSSM may have an effect on student achievement, Schmidt and Houang (2012) Bloom developed his hierarchy of cognitive domains to be used during the planning of instruction to draw attention to the behaviors students would express in relation to the curriculum (Reeves, 1990 or not the student knows, uses, or integrates their content knowledge (Enright, Allen, & Kim, 1993). The complexity analysis study of NAEP science questions included both judgments of difficulty and cognitive demand needed for answering the questions (Enright, Allen & Kim, 1993).…”

“…The complexity analysis study of NAEP science questions included both judgments of difficulty and cognitive demand needed for answering the questions (Enright, Allen & Kim, 1993). Several frameworks that are discussed within this literature review were used to determine question difficulty and cognitive complexity (Bloom, 1956;Glaser, Lesgold, & Lajoie, 1987;Messick, 1984).…”

“…Several frameworks that are discussed within this literature review were used to determine question difficulty and cognitive complexity (Bloom, 1956;Glaser, Lesgold, & Lajoie, 1987;Messick, 1984). Enright et al (1993) posited that item complexity and the cognitive level needed to answer questions should be somehow correlated. However, they found that "there was no evidence that the items' cognitive demand was related to item difficulty" (Enright et al, 1993, pp.…”

“…The study by Enright et al (1993) is important to this study because the complexity level framework was driven initially by Bloom's taxonomy (1956). The framework for assessing cognitive complexity in this study is also based upon Bloom's taxonomy (1956) since the taxonomy and educational objectives identified by Bloom are also embedded in the Webb depth-of-knowledge framework.…”

“…They cited a hierarchy from Enright et al (1993) along with their own semantics for a system for analyzing semantic and cognitive demand of computer-generated questions. The portion of the Benton et al (2004) study that relates to that done by Enright et al (1993) references item difficulty due to the wording utilized in that item, as well as concepts tested (Benton et al, 2004). Additionally, the study by Benton et al (2004) identified a lack in relationship between the items and the concepts being tested; thus, the standards were not strongly represented by the test items.…”