A1 Journal article (refereed)
All types of experience are equal, but some are more equal : The effect of different types of experience on rater severity and rater consistency (2024)

Neittaanmäki, R., & Lamprianou, I. (2024). All types of experience are equal, but some are more equal : The effect of different types of experience on rater severity and rater consistency. Language Testing, OnlineFirst. https://doi.org/10.1177/02655322241239362

JYU authors or editors

Publication details

All authors or editors: Neittaanmäki, Reeta; Lamprianou, Iasonas

Journal or series: Language Testing

ISSN: 0265-5322

eISSN: 1477-0946

Publication year: 2024

Publication date: 09/04/2024

Volume: OnlineFirst

Publisher: SAGE Publications

Publication country: United Kingdom

Publication language: English

DOI: https://doi.org/10.1177/02655322241239362

Publication open access: Openly available

Publication channel open access: Partially open access channel

Publication is parallel published (JYX): https://jyx.jyu.fi/handle/123456789/94521

Abstract

This article focuses on rater severity and consistency and their relation to different types of rater experience over a long period of time. The article is based on longitudinal data collected from 2009 to 2019 from the second language Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. The study investigated whether rater severity and consistency are affected differently by different types of rater experience and by skipping rating sessions. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analyzed using the Many-Facets Rasch model and generalized linear mixed models. The results showed that when the raters gained more rating experience, they became slightly more lenient, but different types of experience had quantitatively different magnitudes of impact. In addition, skipping rating sessions, and in that way disconnecting from the rater community, increased the likelihood of a rater to be inconsistent. Finally, we provide methodological recommendations for future research and consider implications for practice.

Keywords: evaluation; languages; language skills; Finnish language

Free keywords: generalized linear mixed models; Many-Facets Rasch model; rater consistency; rater experience; rater severity; speaking assessment

Contributing organizations

Related projects

Ministry reporting: Yes

Preliminary JUFO rating: 2