Experienced but detached from reality: Theorizing and operationalizing the relationship between experience and rater effects

Lamprianou, Iasonas; Tsagari, Dina; Kyriakou, Nansia

doi:10.1016/j.asw.2023.100713

Assessing Writing

2023

DOI: 10.1016/j.asw.2023.100713

|View full text |Cite

Experienced but detached from reality: Theorizing and operationalizing the relationship between experience and rater effects

Iasonas Lamprianou

Dina Tsagari

Nansia Kyriakou

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2024

Publication Types

Select...

Article2

Relationship

Self Cite0

Independent2

Authors

Journals

Cited by 2 publications

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

All types of experience are equal, but some are more equal: The effect of different types of experience on rater severity and rater consistency

Neittaanmäki,

Lamprianou

2024

Language Testing

View full text Add to dashboard Cite

This article focuses on rater severity and consistency and their relation to different types of rater experience over a long period of time. The article is based on longitudinal data collected from 2009 to 2019 from the second language Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. The study investigated whether rater severity and consistency are affected differently by different types of rater experience and by skipping rating sessions. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analyzed using the Many-Facets Rasch model and generalized linear mixed models. The results showed that when the raters gained more rating experience, they became slightly more lenient, but different types of experience had quantitatively different magnitudes of impact. In addition, skipping rating sessions, and in that way disconnecting from the rater community, increased the likelihood of a rater to be inconsistent. Finally, we provide methodological recommendations for future research and consider implications for practice.

show abstract

All types of experience are equal, but some are more equal: The effect of different types of experience on rater severity and rater consistency

Neittaanmäki,

Lamprianou

2024

Language Testing

View full text Add to dashboard Cite

show abstract

Communal factors in rater severity and consistency over time in high-stakes oral assessment

Neittaanmäki,

Lamprianou

2024

Language Testing

View full text Add to dashboard Cite

This article focuses on rater severity and consistency and their relation to major changes in the rating system in a high-stakes testing context. The study is based on longitudinal data collected from 2009 to 2019 from the second language (L2) Finnishspeaking subtest in the National Certificates of Language Proficiency in Finland. We investigated whether rater severity and consistency changed over that period and whether the changes could be explained by major changes in the rating system, such as the change of lead examiner, the modus of rating and training (on-site or remote), and the composition of the rater group. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analysed using the Many-Facets Rasch model and generalized linear mixed models. The analyses indicated that raters as a group became somewhat more lenient over time. In addition, the results showed that the rater community and its practices, the lead examiners, and the modus of rating and training can influence the rating behaviour. Finally, we elaborate on implications for both research and practice.

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

Experienced but detached from reality: Theorizing and operationalizing the relationship between experience and rater effects

Cited by 2 publications

References 47 publications

All types of experience are equal, but some are more equal: The effect of different types of experience on rater severity and rater consistency

All types of experience are equal, but some are more equal: The effect of different types of experience on rater severity and rater consistency

Communal factors in rater severity and consistency over time in high-stakes oral assessment

Contact Info

Product

Resources

About