Source inclusion in synthesis writing: an NLP approach to understanding argumentation, sourcing, and essay quality

Abstract

Synthesis writing is widely taught across domains and serves as an important means of assessing writing ability, text comprehension, and content learning. Synthesis writing differs from other types of writing in terms of both cognitive and task demands because it requires writers to integrate information across source materials. However, little is known about how integration of source material may influence overall writing quality for synthesis tasks. This study examined approximately 900 source-based essays written in response to four different synthesis prompts which instructed writers to use information from the sources to illustrate and support their arguments and clearly indicate from which sources they were drawing (i.e., citation use). The essays were then scored by expert raters for holistic quality, argumentation, and source use/inferencing. Hand-crafted natural language processing (NLP) features and pre-existing NLP tools were used to examine semantic and keyword overlap between the essays and the source texts, plagiarism from the source texts, and instances of source citation and quoting. These variables along with text length and prompt were then used to predict essays scores. Results reported strong models for predicting human ratings that explained between 47 and 52% of the variance in scores. The results indicate that text length was the strongest predictor of score but also that more successful writers include stronger, semantically-related information from the source, provide more citations and do so later in the text, and copy less from the text. This work introduces the use of NLP techniques to assess source integration, provides details on the types of source integration used by writers, and highlights the effects of source integration on writing quality.

Introduction to the special issue on synthesis tasks: where reading and writing meet

“…Over the last decades, technological developments have led to a fast increase in tools that aim to help writers to develop their writing process and to produce better texts (Limpo et al, 2020). As mentioned before, the study by Crossley et al (2021) of this special issue opens up possibilities for providing students with automatic feedback on their source integration based on NLP-features.…”

“…The issue closes with a study on the automatic assessment of synthesis texts. Crossley et al (2021) introduce the use of natural language processing techniques to assess source integration in synthesis writing. If automatic text scoring via NLP predicts human assessments of these texts, new venues for research are open.…”

“…Various automated essay scoring tools have been developed that reach a level of accuracy in essay scoring that is as accurate as expert human raters (Allen et al, 2021). Crossley et al (2021) contributed to this special issue with a study that used an NLP-approach to provide automatic assessment of source integration and citation reliability. The study's contribution lies in the exploration of the added value of automated text analysis or automatic NLP-features for research purposes and for learning and teaching source-based texts.…”

“…The current special issue offers a palette of studies in which participants perform tasks involving writing and reading. This may be the case because participants need to master a functional writing task (Konstantinidou et al, 2022), write something based on sources (Crossley et al, 2021), or, the most common task in this issue, write a synthesis text, either to demonstrate understanding of sources or to demonstrate their skill in writing (cf. other studies).…”

Writing Quality Predictive Modeling: Integrating Register-Related Factors

“…Though generally considered a desirable rhetorical move (Deane et al, 2013), inappropriate and excessive textual borrowing does not necessarily reflect students' actual language skills, making assessment work with source-based writing more difficult (Weigle & Parker, 2012). The writing rubric we used did not address excessive source borrowing or uncited textual content (see limitations below), but this weakness is not uncommon in writing rubrics for late elementary-aged students (see S. Crossley et al, 2021).…”