A1 Journal article (refereed)
Communal factors in rater severity and consistency over time in high-stakes oral assessment (2024)
Neittaanmäki, R., & Lamprianou, I. (2024). Communal factors in rater severity and consistency over time in high-stakes oral assessment. Language Testing, OnlineFirst. https://doi.org/10.1177/02655322241239363
JYU authors or editors
Publication details
All authors or editors: Neittaanmäki, Reeta; Lamprianou, Iasonas
Journal or series: Language Testing
ISSN: 0265-5322
eISSN: 1477-0946
Publication year: 2024
Publication date: 09/04/2024
Volume: OnlineFirst
Publisher: SAGE Publications
Publication country: United Kingdom
Publication language: English
DOI: https://doi.org/10.1177/02655322241239363
Publication open access: Openly available
Publication channel open access: Partially open access channel
Publication is parallel published (JYX): https://jyx.jyu.fi/handle/123456789/94601
Abstract
This article focuses on rater severity and consistency and their relation to major changes in the rating system in a high-stakes testing context. The study is based on longitudinal data collected from 2009 to 2019 from the second language (L2) Finnishspeaking subtest in the National Certificates of Language Proficiency in Finland. We investigated whether rater severity and consistency changed over that period and whether the changes could be explained by major changes in the rating system, such as the change of lead examiner, the modus of rating and training (on-site or remote), and the composition of the rater group. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analysed using the Many-Facets Rasch model and generalized linear mixed models. The analyses indicated that raters as a group became somewhat more lenient over time. In addition, the results showed that the rater community and its practices, the lead examiners, and the modus of rating and training can influence the rating behaviour. Finally, we elaborate on implications for both research and practice.
Keywords: evaluation; languages; language skills
Free keywords: generalized linear mixed model; lead examiner; Many-Facets Rasch model; on-site rating; rater consistency; rater severity; remote rating; speaking assessment
Contributing organizations
Ministry reporting: Yes
VIRTA submission year: 2024
Preliminary JUFO rating: 2