A1 Journal article (refereed)
Communal factors in rater severity and consistency over time in high-stakes oral assessment (2024)

Neittaanmäki, R., & Lamprianou, I. (2024). Communal factors in rater severity and consistency over time in high-stakes oral assessment. Language Testing, OnlineFirst. https://doi.org/10.1177/02655322241239363

JYU authors or editors

Publication details

All authors or editors: Neittaanmäki, Reeta; Lamprianou, Iasonas

Journal or series: Language Testing

ISSN: 0265-5322

eISSN: 1477-0946

Publication year: 2024

Publication date: 09/04/2024

Volume: OnlineFirst

Publisher: SAGE Publications

Publication country: United Kingdom

Publication language: English

DOI: https://doi.org/10.1177/02655322241239363

Publication open access: Openly available

Publication channel open access: Partially open access channel

Publication is parallel published (JYX): https://jyx.jyu.fi/handle/123456789/94601

Abstract

This article focuses on rater severity and consistency and their relation to major changes in the rating system in a high-stakes testing context. The study is based on longitudinal data collected from 2009 to 2019 from the second language (L2) Finnishspeaking subtest in the National Certificates of Language Proficiency in Finland. We investigated whether rater severity and consistency changed over that period and whether the changes could be explained by major changes in the rating system, such as the change of lead examiner, the modus of rating and training (on-site or remote), and the composition of the rater group. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analysed using the Many-Facets Rasch model and generalized linear mixed models. The analyses indicated that raters as a group became somewhat more lenient over time. In addition, the results showed that the rater community and its practices, the lead examiners, and the modus of rating and training can influence the rating behaviour. Finally, we elaborate on implications for both research and practice.

Keywords: evaluation; languages; language skills

Free keywords: generalized linear mixed model; lead examiner; Many-Facets Rasch model; on-site rating; rater consistency; rater severity; remote rating; speaking assessment

Contributing organizations

Ministry reporting: Yes

Preliminary JUFO rating: 2