Examining Rater Performance on the CELBAN Speaking: A Many-Facets Rasch Measurement Analysis

Abstract

Internationally educated nurses’ (IENs) English language proficiency is critical to professional licensure as communication is a key competency for safe practice. The Canadian English Language Benchmark Assessment for Nurses (CELBAN) is Canada’s only Canadian Language Benchmarks (CLB) referenced examination used in the context of healthcare regulation. This high-stakes assessment claims proof of proficiency for IENs seeking licensure in Canada and a measure of public safety for nursing regulators. Understanding the quality of rater performance when examination results are used for high-stakes decisions is crucial to maintaining speaking test quality as it involves judgement, and thus requires strong reliability evidence (Koizumi et al., 2017). This study examined rater performance on the CELBAN Speaking component using a Many-Facets Rasch Measurement (MFRM). Specifically, this study identified CELBAN rater reliability in terms of consistency and severity, rating bias, and use of rating scale. The study was based on a sample of 115 raters across eight test sites in Canada and results on 2698 examinations across four parallel versions. Findings demonstrated relatively high inter-rater reliability and intra-rater reliability, and that CLB-based speaking descriptors (CLB 6-9) provided sufficient information for raters to discriminate examinees’ oral proficiency. There was no influence of test site or test version, offering validity evidence to support test use for high-stakes purposes. Grammar, among the eight speaking criteria, was identified as the most difficult criterion on the scale, and the one demonstrating most rater bias. This study highlights the value of MFRM analysis in rater performance research with implications for rater training. This study is one of the first research studies using MFRM with a CLB-referenced high-stakes assessment within the Canadian context.

Raters’ Assessment Quality in Measuring Teachers’ Competency in Classroom Assessment: Application of Many Facet Rasch Model

“…The analysis by using MFRM has gained much attention from researchers and has been widely used in language testing, education and psychological measurement (Barkaoui, 2013;Linacre, 1994). MFRM is also widely used in other areas such as study in nutrition by Sunjaya et al (2020), research to determine the quality of rater's judgement in The Canadian English Language Benchmark Assessment for Nurses (CELBAN) by Wang et al (2021) and research to analyse the content validity for Computerized Testlet Instrument to Measure Chemical Literacy Capabilities by Fahmina et al (2019).…”

The Rater Performance Categorization System (RPCS) for Intensive English Programs

“…They also suggested using larger data in similar studies as they thought this outcome could be due to the small sample data they utilized. Wang, et al, (2020) examined the raters' performance in the Canadian English Language Benchmark Assessment for Nurses (CELBAN) exam speaking component in terms of raters consistency and severity, use of rating scales, and rating bias using Many-facets Rasch Measurement (Linacre & Wright, 1989). They used 115 raters and 2,698 examinations in four parallel forms.…”

“…It is thought by the author of this paper that there are some reasons behind it. To elaborate, as mentioned earlier most popular rater monitoring systems currently available are based on Many Facets Rasch Measurement (MFRM; Wang et al, 2020;Myford & Wolfe, 2009;Wigglesworth, 1993;Davis, 2016), Bayesian approach (Cao, et al, 2010), Rasch Partial Credit model (Wang, et al, 2017), Hierarchical rater model (DeCarlo, et al, 2011), and automated scoring engines (Shin, et al, 2019). These monitoring systems provide the administrators with highly detailed and robust systems which can be attained by using complex mathematical models, and methods like the Bayesian method (Cao, et al, 2010), Maximum likelihood estimation (Shin, et al, 2019), log-ratio test (Wang, et al, 2017), time facet model (Myford & Wolfe, 2009), Signal detection rater model (DeCarlo, et al, 2011), generalized partial credit model (DeCarlo, et al, 2011), mixedeffects ordinal probit model (Shin, et al, 2019) and some specialized software like Facets (Linacre, 2014), Winsteps (Linacre, 2018), Stata (Stata Corp., 2013, latent gold (Vermunt & Magidson, 2005), glam (Rabe-Hesketh, et al, 2004), and WinBUGS (Lunn, et al, 2009).…”