Exploring Relationships Between Writing and Broader Outcomes With Automated Writing Evaluation

Abstract

No significant body of research examines writing achievement and the specific skills and knowledge in the writing domain for postsecondary (college) students in the U.S., even though many at-risk students lack the prerequisite writing skills required to persist in their education. This paper addresses this gap through a novel exploratory study examining how automated writing evaluation (AWE) can inform our understanding of the relationship between postsecondary writing skill and broader indicators of college success. The exploratory study presented in this paper was conducted using test-taker essays from a standardized writing assessment of postsecondary student learning outcomes. Findings showed that for the essays, AWE features were found to be predictors of broader outcomes measures: college success indicators and learning outcomes measures. Study findings expose AWE's potential to support educational analytics --i.e., relationships between writing skill and broader outcomes -moving AWE beyond writing assessment and instructional use cases.

I Can Expand My Imagination: The Secondary Students’ Narrative in Creative Writing English Through Padlet

“…Consequently, the assessment of students' writing proficiency, encompassing sophisticated competencies that hold significance for learners. It is line with Burstein et al, (2020) argue that the attainment of writing proficiency is contingent upon an interplay of socio-cognitive factors, encompassing proficiency in writing techniques, cognitive aptitude, engagement, and capacity for collaborative work.…”

Arriving at the everyday: building the NUWPArc public digital writing program archive

“…Despite noble goals touted by many writing programs such as developing "clear and dynamic writers for public service" and demonstrating "the powers of careful thinking and critical analysis," 64 student "success" (often characterized by retention, graduation, and career preparedness metrics) is situated in the eyes of many university administrators and educational policymakers as the primary measurement goal of program assessment (i.e., how well is this program preparing students to "succeed?"). 65 While funding and time is a challenge for many WPAs, punctive agents have tools and resources that allow them to "address this gap" in informing "our understanding of the relationship between postsecondary writing skill and broader indicators of college success" (Burstein et al, 2017). 66 Digital archives can be used as tools to capture dynamic patterns of emergence in assessment during the period covered by the archive given that, in writing evaluation, assessment has evolved from testing first designed to identify "remedial" writers, to place them into courses, and to grade or score writers' performance in classrooms.…”

“…I suggest that part of what allows punctive agents to recontour the field is an overwrought attention to the importance of belief-particularly since beliefs are mutable and can change with exposure to new kinds of disciplinary training.While we argue about what beliefs establish our authority to enact our values, punctive agents offer tools that promise efficiency and results. One of the primary tools offered by punctive agents for use at and across writing sites is automated writing evaluation (AWE), which is based in NLP Burstein et al (2017). describe the "real-time, dynamic nature of NLP-based AWE" as affording researchers the ability to "explore linguistic features and skill relationships across a range of writing genres in postsecondary education" that can "provide educational analytics that could be informative for various stakeholders" (p. 101).…”