The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap

Abstract

This article introduces the second version of the Tool for the Automatic Analysis of Cohesion (TAACO 2.0). Like its predecessor, TAACO 2.0 is a freely available text analysis tool that works on the Windows, Mac, and Linux operating systems; is housed on a user's hard drive; is easy to use; and allows for batch processing of text files. TAACO 2.0 includes all the original indices reported for TAACO 1.0, but it adds a number of new indices related to local and global cohesion at the semantic level, reported by latent semantic analysis, latent Dirichlet allocation, and word2vec. The tool also includes a source overlap feature, which calculates lexical and semantic overlap between a source and a response text (i.e., cohesion between the two texts based measures of text relatedness). In the first study in this article, we examined the effects that cohesion features, prompt, essay elaboration, and enhanced cohesion had on expert ratings of text coherence, finding that global semantic similarity as reported by word2vec was an important predictor of coherence ratings. A second study was conducted to examine the source and response indices. In this study we examined whether source overlap between the speaking samples found in the TOEFL-iBT integrated speaking tasks and the responses produced by test-takers was predictive of human ratings of speaking proficiency. The results indicated that the percentage of keywords found in both the source and response and the similarity between the source document and the response, as reported by word2vec, were significant predictors of speaking quality. Combined, these findings help validate the new indices reported for TAACO 2.0. Keywords Cohesion. Coherence. Natural language processing. Essay quality. Speaking proficiency Cohesion generally refers to text elements that help readers facilitate comprehension (Graesser, McNamara, & Louwerse, 2003). These elements can include argument overlap across sentences and paragraphs, the use of connectives to link sentence and paragraphs, and casual relationships within a text. Cohesion can be categorized at a base level as either local or global, where local cohesion refers to textual links between sentences, and global cohesion identifies links between larger text segments such as paragraphs (Givón, 1995; Kintsch, 1995). Measuring text cohesion is important because it can help identify text that is more difficult to comprehend, text genres, and writing quality

To What Extent Are Multiword Sequences Associated With Oral Fluency?

“…In similar findings to those of the current study, Crossley et al. () reported that advanced speakers used a number of key‐phrases (i.e., four‐word sequences from the source text) in their speech, and that expert raters judged such speech samples to be proficient.…”

“…First, the results underlined the important influence of task instructions on use of task‐specific MWSs, supporting the well‐documented learner behavior of text mining (i.e., transferring MWSs from input texts to L2 output; Boers et al., ; Hoang & Boers, ). The observed tendency of advanced speakers to use a great number of task‐specific expressions accords with previous research exploring cohesion (or keyword and key‐phrase overlap) between the task prompt and the speaker response and its impact on expert ratings of oral proficiency in the TOEFL‐iBT integrated tasks (Crossley, Kyle, & Dascalu, ). In similar findings to those of the current study, Crossley et al.…”

The Effects of Read-aloud Assistance on Second Language Oral Fluency in Text Summary Speech

“…These indices tap into the extent to which characteristic words and n-grams from the source text are used in the speech transcript. TAACO 2.0 identifies single-and multi-word keywords using the news and magazine sections of the Corpus of Contemporary American English (COCA) as a reference corpus (for a detailed description, see Crossley et al, 2019).…”

Assisting students’ writing with computer-based concept map feedback: A validation study of the CohViz feedback system

“…To investigate the validity of Coh-Viz, we benchmarked the three structural indicators obtained by CohViz to convergent and divergent measures of text cohesion commonly used in well-established assessment systems for writing quality [36,46]. Convergent validity can be assessed by comparing the fragments as the central measure of text cohesion in CohViz to other text cohesion measures (e.g., argument overlap, LSA-dependent cohesion measures) [39]. Divergent validity can be assessed by comparing CohViz to divergent measures of text cohesion (e.g., syntactic or lexical complexity as indicators of writing quality).…”

“…Regarding the validity of CohViz, we assessed the convergent and divergent validity of the CohViz system with automated measures from well-established assessment systems [36][37][38][39], as used for example by the well-known system Coh-Metrix [40]. As CohViz was predominantly designed to provide feedback on the cohesion of students' writing, we focused on the number of fragments as the central indicator of cohesion provided by CohViz and compared it with convergent (i.e., argument overlap, semantic overlap) and divergent (i.e., syntactic complexity, lexical diversity, and word concreteness) linguistic features of text cohesion.…”