The Quality of Content Analyses of State Student Achievement Tests and Content Standards

Abstract

This article examines the reliability of content analyses of state student achievement tests and state content standards. We use data from two states in three grades in mathematics and English language arts and reading to explore differences by state, content area, grade level, and document type. Using a generalizability framework, we find that reliabilities for four coders are generally greater than .80. For the two problematic reliabilities, they are partly explained by an odd rater out. We conclude that the content analysis procedures, when used with at least five raters, provide reliable information to researchers, policymakers, and practitioners about the content of assessments and standards.

Application of Evidence-Centered Assessment Design to the Advanced Placement Redesign: A Graphic Restatement

“…A possible approach to characterizing content validity, at least within an accountability context, is by means of the alignment of content standards and the test. Methodologies to that effect have been developed (Porter, Polikoff, Zeidner, & Smithson, 2008;Webb, 2007). While alignment, or content validity, is important, it is not sufficient by itself to characterize the validity of scores produced by a test, as has been suggested recently Lissitz and Samuelsen (2007).…”

Examining alignment : national and local assessments and the common core state standards in mathematics.

“…Countering these finding, a study conducted by the Center on Education Policy (CEP) (Chudowsky & Chudowsky, 2011) reported an This approach compared mathematics content and cognitive demand by dividing them into 16 content areas. Using a formula for alignment index developed by Porter (2002) and deemed reliable through multiple studies (Porter, Polikoff, Zeidner, & Smithson, 2008;Porter, 2002;Polikoff, Porter, & Smithson, 2009;and, Porter, Kirst, Osthoff, Smithson, & Schneider, 1993), the mathematics standards revealed an average of 20% alignment between state standards and CCSSM standards in eighth-grade mathematics (Porter et al, 2011, p. 111). The greatest difference in content standards for individual states compared to the CCSSM across all grades involved the number of standards using…”

“…Therefore the reliability of the instrument relies heavily upon the degrees of agreement of the raters. Porter, Polikoff, Zeidner, and Smithson (2008) performed a study on rater reliability in alignment studies. The study used the SEC as the alignment tool; however, the focus of the study was to determine the number of content experts to use for coding versus the reliability of the instrument itself.…”

“…Textbook publishers (Buckendahl, Plake, Impara, & Irwin, 2000), college faculty and high school teachers (Herman, Webb, & Zuniga, 2005) each rated the alignment of standards and assessments differently. Porter et al (2008) indicated that training and raters' backgrounds had a profound effect on rater reliability. In addition, the number of raters should be considered for inter-rater reliability.…”

“…In addition, the number of raters should be considered for inter-rater reliability. No definitive number of individuals for coding has been identified as the most appropriate for the highest level of reliability; however, five or six coders would be sufficient for using the model for alignment (LaMarca, Redfield, Winter & Bailey, 2000;Porter et al, 2008;Webb 1997aWebb & 1999 …”