Comparing holistic and analytic marking methods in assessing speech act production in L2 Chinese

Abstract

This study compared holistic and analytic marking methods for their effects on parameter estimation (of examinees, raters, and items) and rater cognition in assessing speech act production in L2 Chinese. Seventy American learners of Chinese completed an oral Discourse Completion Test assessing requests and refusals. Four native Chinese raters evaluated the examinees’ oral productions using two four-point rating scales. The holistic scale simultaneously included the following five dimensions: communicative function, prosody, fluency, appropriateness, and grammaticality; the analytic scale included sub-scales to examine each of the five dimensions. The raters scored the dataset twice with the two marking methods, respectively, and with counterbalanced order. They also verbalized their scoring rationale after performing each rating. Results revealed that both marking methods led to high reliability and produced scores with high correlation; however, analytic marking possessed better assessment quality in terms of higher reliability and measurement precision, higher percentages of Rasch model fit for examinees and items, and more balanced reference to rating criteria among raters during the scoring process.

Comparing two formats of data-driven rating scales for classroom assessment of pragmatic performance with roleplays

“…This criterion does not account for prosody, which has started to be integrated into the analysis of speech perception and production by learners in L2 pragmatics (see Kang & Kermad, 2019 for a review). In fact, there has been some uptake in measurement-driven scales developed for pragmatic performance (Li et al, 2023). In addition, as the roleplay test target includes lower-proficiency learners, scale descriptors catering to limited production (e.g., the binary decision of “with few sentences” in Hirai & Koizumi, 2013), could be beneficial in assessing the language use of those who produce very little content, albeit accurately.…”

“…It is only recently that there has been empirical research in the domain of pragmatics assessment on scale function (Li et al, 2019(Li et al, , 2023. As the conceptualization of pragmatic competence has evolved to involve the ability to use the knowledge of formfunction-context mapping in an adaptive manner in interaction (Taguchi, 2019;Taguchi & Li, 2020), more attention is now being paid to roleplay as an assessment tool, which seeks to elicit interactive and contextualized discourse for measuring learners' online pragmatic competence (Roever, 2011).…”