A1 Alkuperäisartikkeli tieteellisessä aikakauslehdessä
Communal factors in rater severity and consistency over time in high-stakes oral assessment (2024)
Neittaanmäki, R., & Lamprianou, I. (2024). Communal factors in rater severity and consistency over time in high-stakes oral assessment. Language Testing, OnlineFirst. https://doi.org/10.1177/02655322241239363
JYU-tekijät tai -toimittajat
Julkaisun tiedot
Julkaisun kaikki tekijät tai toimittajat: Neittaanmäki, Reeta; Lamprianou, Iasonas
Lehti tai sarja: Language Testing
ISSN: 0265-5322
eISSN: 1477-0946
Julkaisuvuosi: 2024
Ilmestymispäivä: 09.04.2024
Volyymi: OnlineFirst
Kustantaja: SAGE Publications
Julkaisumaa: Britannia
Julkaisun kieli: englanti
DOI: https://doi.org/10.1177/02655322241239363
Julkaisun avoin saatavuus: Avoimesti saatavilla
Julkaisukanavan avoin saatavuus: Osittain avoin julkaisukanava
Julkaisu on rinnakkaistallennettu (JYX): https://jyx.jyu.fi/handle/123456789/94601
Tiivistelmä
This article focuses on rater severity and consistency and their relation to major changes in the rating system in a high-stakes testing context. The study is based on longitudinal data collected from 2009 to 2019 from the second language (L2) Finnishspeaking subtest in the National Certificates of Language Proficiency in Finland. We investigated whether rater severity and consistency changed over that period and whether the changes could be explained by major changes in the rating system, such as the change of lead examiner, the modus of rating and training (on-site or remote), and the composition of the rater group. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analysed using the Many-Facets Rasch model and generalized linear mixed models. The analyses indicated that raters as a group became somewhat more lenient over time. In addition, the results showed that the rater community and its practices, the lead examiners, and the modus of rating and training can influence the rating behaviour. Finally, we elaborate on implications for both research and practice.
YSO-asiasanat: arviointi; kieli ja kielet; kielitaito
Vapaat asiasanat: generalized linear mixed model; lead examiner; Many-Facets Rasch model; on-site rating; rater consistency; rater severity; remote rating; speaking assessment
Liittyvät organisaatiot
OKM-raportointi: Kyllä
Alustava JUFO-taso: 2
