Instructional Sensitivity of a State's Standards-Based Assessment

Abstract

The accuracy of achievement test score inferences largely depends on the sensitivity of scores to instruction focused on tested objectives. Sensitivity requirements are particularly challenging for standards-based assessments because a variety of plausible instructional differences across classrooms must be detected. For this study, we developed a new method for capturing the alignment between how teachers bring standards to life in their classrooms and how the standards are defined on a test. Teachers were asked to report the degree to which they emphasized the state's academic standards, and to describe how they taught certain objectives from the standards. Two curriculum experts judged the alignment between how teachers brought the objectives to life in their classrooms and how the objectives were operationalized on the state test. Emphasis alone did not account for achievement differences among classrooms. The best predictors of classroom achievement were the match between how the standards were taught and tested, and the interaction between emphasis and match, indicating that test scores were sensitive to instruction of the standards, but in a narrow sense.

Grading as a Reform Effort: Do Standards‐Based Grades Converge With Test Scores?

“…Similarly, Conley () theorized that the grade–test score relationship may be weakened due to variations in the way that teachers operationalize learning goals and in their requirements for a “proficient” grade. D'Agostino, Welsh, & Corson () confirmed this theory by examining the degree of alignment between operationalization of state mathematics standards on a state test and in classrooms. They then used degree of alignment to predict mathematics performance and found both that teachers varied in the ways that they implemented the standards and that the degree of alignment predicted student achievement.…”

Opportunity to Learn What Is on the Test and Performance on the Test

“…That is, alignment between the enacted and intended curriculum cannot be expected to correlate highly with student achievement if the test fails to be aligned with the respective content standards. In addition, the instructional sensitivity of assessments used to detect the influence of OTL on achievement typically remains an unexamined assumption among researchers (D’Agostino, Welsh, & Corson, 2007). Another limitation in the available data on OTL related to the content of instruction is the paucity of research involving SWDs, especially those students receiving all their instruction in general education classrooms.…”

Modeling Instructional Sensitivity Using a Longitudinal Multilevel Differential Item Functioning Approach

“…Empirical approaches to quantify instructional sensitivity emerged in the context of criterion‐referenced testing (e.g., Popham, ). Polikoff () recently categorized the numerous approaches according to the evidence used: (1) expert judgment (e.g., Popham, ), (2) instructional measures (e.g., D'Agostino et al., ), or (3) item statistics (e.g., Cox & Vargas, ; Linn & Harnisch, ). To date, the validity of ratings on instructional sensitivity seems rather unclear, and it still remains unknown which instructional measures really are relevant for the investigation of instructional sensitivity (Polikoff, ).…”

“…Instructional sensitivity is defined as the psychometric property of a test or a single item to be sensitive to instruction (Polikoff, 2010). The concept relates to the extent to which tests and items are capable of detecting effects of the implemented curriculum (Travers & Westbury, 1989), in particular, the content and the quality of instruction (D'Agostino, Welsh, & Corson, 2007). Airasian and Madaus (1983) emphasize instructional sensitivity as an important aspect of construct validity (Cronbach & Meehl, 1955).…”