Construct validity in TOEFL iBT speaking tasks: Insights from natural language processing

Kyle, Kristopher; Crossley, Scott A.; McNamara, Danielle S.

doi:10.1177/0265532215587391

Cited by 35 publications

(25 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In addition, words integrated from the source texts into the responses were highly predictive of speaking quality. Finally, Kyle, Crossley, and McNamara (2016) examined differences between L2 speakers' responses to both independent and integrated speaking tasks and found that independent speaking tasks result in less given information (i.e., information available previously in the discourse). Although studies examining cohesion and spoken discourse in terms of comprehension and processing are infrequent, the existing studies above indicate that cohesion features in discourse, as well as cohesive links between the speaking prompt/source text and a response, are important components of speaking quality.…”

Section: Assessing Discourse Cohesionmentioning

confidence: 99%

The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap

2018

Self Cite

View full text Add to dashboard Cite

This article introduces the second version of the Tool for the Automatic Analysis of Cohesion (TAACO 2.0). Like its predecessor, TAACO 2.0 is a freely available text analysis tool that works on the Windows, Mac, and Linux operating systems; is housed on a user's hard drive; is easy to use; and allows for batch processing of text files. TAACO 2.0 includes all the original indices reported for TAACO 1.0, but it adds a number of new indices related to local and global cohesion at the semantic level, reported by latent semantic analysis, latent Dirichlet allocation, and word2vec. The tool also includes a source overlap feature, which calculates lexical and semantic overlap between a source and a response text (i.e., cohesion between the two texts based measures of text relatedness). In the first study in this article, we examined the effects that cohesion features, prompt, essay elaboration, and enhanced cohesion had on expert ratings of text coherence, finding that global semantic similarity as reported by word2vec was an important predictor of coherence ratings. A second study was conducted to examine the source and response indices. In this study we examined whether source overlap between the speaking samples found in the TOEFL-iBT integrated speaking tasks and the responses produced by test-takers was predictive of human ratings of speaking proficiency. The results indicated that the percentage of keywords found in both the source and response and the similarity between the source document and the response, as reported by word2vec, were significant predictors of speaking quality. Combined, these findings help validate the new indices reported for TAACO 2.0. Keywords Cohesion. Coherence. Natural language processing. Essay quality. Speaking proficiency Cohesion generally refers to text elements that help readers facilitate comprehension (Graesser, McNamara, & Louwerse, 2003). These elements can include argument overlap across sentences and paragraphs, the use of connectives to link sentence and paragraphs, and casual relationships within a text. Cohesion can be categorized at a base level as either local or global, where local cohesion refers to textual links between sentences, and global cohesion identifies links between larger text segments such as paragraphs (Givón, 1995; Kintsch, 1995). Measuring text cohesion is important because it can help identify text that is more difficult to comprehend, text genres, and writing quality

show abstract

Section: Assessing Discourse Cohesionmentioning

confidence: 99%

The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap

2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…Some previous research on rater behavior has demonstrated a considerable amount of rater variability, which is mostly related to raters' characteristics and not the test takers' performance (e.g., Carey, Mannell, & Dunn, 2011;Knoch, 2011). Accordingly, several research studies have been carried out to investigate the accuracy of oral performance assessment tests through research on both rater reliability, which is the degree of agreement among independent raters in assessing the test takers (Khabbazbashi, 2017;Winke & Gass, 2013), and the validity of performancebased tests through construct validity and concurrent validity (Kyle, Crossley & McNamara, 2016). Furthermore, rater training has demonstrated low impact in reducing this variability.…”

Section: Rater Behavior In Oral Performance Assessmentmentioning

confidence: 99%

“…Second, it can help researchers administer rater training programs. Research has shown that rater consistency and rating validity can be increased through training (Kyle, Crossley, & McNamara, 2016). Third, MFRM can help reduce self-inconsistency and increase intra-rater reliability, which increases the fairness of a test, specifically in placement and summative evaluation tests (Gan, 2010).…”

Section: Rater Behavior In Oral Performance Assessmentmentioning

confidence: 99%

Investigating the Effect of Training on Raters’ Bias toward Test Takers in Oral Proficiency Assessment : A FACETS Analysis

Bijani¹,

Khabiri²

2017

The Journal of AsiaTEFL

View full text Add to dashboard Cite

Typically, variability among raters in scoring and their bias is mediated through rater training. However, questions still remain about whether training can affect raters' severity or leniency. Furthermore, few studies have looked at the differences between trained and untrained raters in oral assessment. Oral test scores of 200 test takers rated by 20 raters and were analyzed before and after a training program using the multifaceted Rasch measurement (MFRM). The results demonstrated the constructive impact of training programs in reducing raters' biases and increasing their consistency measures. This study indicated that inexperienced raters benefited more from a training program than experienced raters and thus achieved higher measures of consistency afterward. It also demonstrated a higher biased interaction for test takers on the extreme ends of the oral ability continuum. The findings demonstrated that it is almost impossible to completely eradicate rater variability even through rater training. Therefore, rater training should be viewed as a procedure to establish within-rater consistency rather than between-rater consistency. Since this study showed that inexperienced raters can rate even more reliably than experienced ones after training, there is no evidence whereby decision makers can exclude inexperienced raters solely because of their lack of adequate experience. Consequently, decision makers need to use their budgets for establishing rater training programs for inexperienced raters instead.

show abstract

“…(1) (e.g., Cumming, 2013) (Hirai & Koizumi, 2009;Swain, Huang, Barkaoui, Brooks, & Lapkin, 2009) 3 (Murray, et al, 2012) 3 Brown, Iwashita, & McNamara (2005) 4 TOEFL iBT (e.g., Kyle, Crossley, & McNamara, 2015) Hirai and Koizumi (2008,2013) EBB (Empirically derived, Binary-choice, Boundary-definition)…”

Section: 35mentioning

confidence: 99%

Vol. 19 Supplementary: 20th Anniversary Special Issue

2016

JLTA Journal

View full text Add to dashboard Cite

Construct validity in TOEFL iBT speaking tasks: Insights from natural language processing

Cited by 35 publications

References 28 publications

The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap

The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap

Investigating the Effect of Training on Raters’ Bias toward Test Takers in Oral Proficiency Assessment : A FACETS Analysis

Vol. 19 Supplementary: 20th Anniversary Special Issue

Contact Info

Product

Resources

About