Moving Beyond Total Words Written: The Reliability, Criterion Validity, and Time Cost of Alternate Measures for Curriculum-Based Measurement in Writing

Exploration of New Complexity Metrics for Curriculum-Based Measures of Writing

“…Other measures of complexity, such as holistic and trait-based rating scales and rubrics, generally have correlated weakly to moderately across grade levels ( r = .35–.76, Coker & Ritchey, 2010; r = .36–.37, Gansle, Noell, VanDerHeyden, Naquin, & Slider, 2002; r = .27, Gansle, VanDerHeyden, Noell, Resetar, & Williams, 2006; r = .06–.67, Lembke et al, 2003; r = .50–.60, McMaster et al, 2009; r = −.02 to .63, Tindal & Parker, 1991). Some studies suggest a stronger relation between accurate production measures (e.g., WSC) and qualitative measures in early elementary grades (e.g., Coker & Ritchey, 2010; Lembke et al, 2003), but these correlations appear to decrease by upper elementary grades (Gansle et al, 2006).…”

Assessing Teacher Usability of Written Expression Curriculum-Based Measurement

“…If screening all students, one scoring metric that has been previously studied as a potential time-efficient metric is Correct Punctuation (Amato & Watkins, 2011; Gansle et al, 2002). Although this might be a brief option for scoring and offer efficiency for universal screening, the research is still limited.…”

“…Most WE-CBM research has focused on technical features of a static score with an emphasis on predictive and criterion validity (e.g., Espin et al, 2000; Gansle, VanDerHeyden, Noell, Naquin, & Slider, 2002; Keller-Margulis, Payan, Jaspers, & Brewton, 2016). To date, there is only limited consensus regarding appropriate scoring metrics for screening, and there is evidence of the need for a developmental perspective (e.g., grade level) when determining which scoring metric to use (Jewell & Malecki, 2005; Malecki & Jewell, 2003) as well as a need to consider language background (e.g., Keller-Margulis, Payan, et al, 2016).…”

The Potential for Automated Text Evaluation to Improve the Technical Adequacy of Written Expression Curriculum-Based Measurement

“…Validity studies for WE-CBM largely indicate results in the weak to moderate range (McMaster & Espin, 2007;Romig et al, 2016). Studies of duration typically include 3-, 5-, 7-, and 10-min samples of writing, and generally find that longer durations provide improved technical adequacy (e.g., Espin et al, 2008;Weissenburger & Espin, 2005), but reduce feasibility due to additional administration and scoring time (Espin, Scierka, Skare, & Halverson, 1999;Gansle, Noell, VanDerHeyden, Naquin, & Slider, 2002).…”

“…Production-dependent metrics include the original WE-CBM metrics such as total words written (TWW) and correct word sequences (CWS), and production-independent metrics include variations of these such as percentage correct word sequences (%CWS) to control for variation in the amount of writing produced. The accurate-production metric of correct minus incorrect writing sequences (CIWS) combines accuracy and fluency (Espin et al, 2000;Espin et al, 2008), and although validity findings tend to be higher than for other metrics (Mercer, Martínez, Faust, & Mitchell, 2012;Romig et al, 2016;Weissenburger & Espin, 2005), this metric requires significant time to reliably score and thus may be less feasible for use in universal screening (Espin et al, 1999;Gansle et al, 2002). Work in this area has largely investigated the technical adequacy of individual metrics such as TWW versus CIWS (McMaster & Espin, 2007); however, given that these metrics, when calculated on the same samples, are moderately to highly correlated, composites based on multiple WE-CBM metrics would likely improve reliability and validity (Codding, Petscher, & Truckenmiller, 2015;Espin et al, 1999).…”