Toward a New Theory of Writing Assessment

Abstract

Toward a New Theory of Writing Assessment Many composition teachers and scholars feel frustrated by, cut off from, or otherwise uninterested in the subject of writing assessment-especially assessment that takes place outside of the classroom for purposes of placement, exit, or program evaluation. This distrust and estrangement are understandable, given the highly technical aspects of much discourse about writing assessment. For the most part, writing assessment has been developed, constructed, and privatized by the measurement community as a technological apparatus whose inner workings are known only to those with specialized knowledge. Consequently, English professionals have been made to feel inadequate and naive by considerations of technical concepts like validity and reliability. At the same time, teachers have remained skeptical (and rightly so) of assessment practices that do not reflect the values important to an understanding of how people learn to read and write. It does not take a measurement specialist to realize that many writing assessment procedures have missed the mark in examining students' writing ability. At the core of this inability to communicate are basic theoretical differences between the measurement and composition communities (White, "Language"). Writing assessment procedures, as they have been traditionally constructed, are designed to produce reliable (that is, consistent) numerical scores of individual student papers from independent judges. Traditional writing assessment practices are based upon classical test theory, with roots in a positivist epistemology that assumes "that there exists a reality out there, driven by immutable natural laws" (Guba 19). The assumption is that student ability in writing, as in anything else, is a fixed, Brian Huot teaches graduate and undergraduate courses on writing at the University of Louisville, where he directs the composition program. He is coeditor of Assessing Writing, the only journal devoted to writing assessment. This essay culminates several years of thinking, talking, and writing about the need to articulate the theories behind writing assessment practices.

English Writing Strategies of Korean Students: Exploring Written Texts and Interviews with the Teacher

“…It was not easy to discover any significant variation in scores between lower and higher ability students across task types. Therefore, existing studies tend to contradict each other's findings; the issues are still open to further research in difficulty level of tasks [13].…”

Comparative appraisal of expressive artifacts

“…In the literature of writing assessment, researchers identified problems with validity when students were asked to produce assessment materials that were not congruent with what instructors valued as good writing [11,17]. For example, writing instructors might believe that good writing requires revision, and yet students would write under timed test conditions for assessment.…”

“…From this set of varied assessments, the ETS researchers analyzed rater comments to derive five broad areas that captured most criteria variously employed by the raters: Ideas, form, flavor (style), mechanics, and wording [7]. To decrease variability of the sort described in the ETS report, writing assessment researchers developed holistic scoring methods based on standardized rubrics that formalize a small set of generalized criteria such as those isolated in the ETS study, supported by rater training sessions in which applying the rubric consistently is emphasized [11]. In the U.S., such methods have been widely adopted for both national (such as Advanced Placement exams) and institutional testing purposes [4,6,18].…”

“…The assessment process, including development and implementation of localized procedures, becomes a means of defining, debating, and articulating those values for a particular instructional community [11,4]. While some proposals for localized assessment argue for getting rid of rubrics entirely, contending that they thwart the recognition of imaginative solutions to writing problems, others retain the structure of rubrics in a more flexible, contextspecific manner [5].…”

Assessing Assessment

“…12 As Huot points out, these early exams did not quite fit the experience of students and teachers in the classroom. 13 This brought about a call for assessment that moved away from focusing on reliability expressed as intercoder reliability and moving more toward a measure that reflected validity. 14 This movement allowed raters to focus on elements such as context and localized standards set by stakeholders and teachers.…”