A1 Journal article (refereed)
All types of experience are equal, but some are more equal : The effect of different types of experience on rater severity and rater consistency (2024)
Neittaanmäki, R., & Lamprianou, I. (2024). All types of experience are equal, but some are more equal : The effect of different types of experience on rater severity and rater consistency. Language Testing, OnlineFirst. https://doi.org/10.1177/02655322241239362
JYU authors or editors
Publication details
All authors or editors: Neittaanmäki, Reeta; Lamprianou, Iasonas
Journal or series: Language Testing
ISSN: 0265-5322
eISSN: 1477-0946
Publication year: 2024
Publication date: 09/04/2024
Volume: OnlineFirst
Publisher: SAGE Publications
Publication country: United Kingdom
Publication language: English
DOI: https://doi.org/10.1177/02655322241239362
Publication open access: Openly available
Publication channel open access: Partially open access channel
Publication is parallel published (JYX): https://jyx.jyu.fi/handle/123456789/94521
Abstract
This article focuses on rater severity and consistency and their relation to different types of rater experience over a long period of time. The article is based on longitudinal data collected from 2009 to 2019 from the second language Finnish speaking subtest in the National Certificates of Language Proficiency in Finland. The study investigated whether rater severity and consistency are affected differently by different types of rater experience and by skipping rating sessions. The data consisted of 45 rating sessions with 104 raters and 59,899 examinees and were analyzed using the Many-Facets Rasch model and generalized linear mixed models. The results showed that when the raters gained more rating experience, they became slightly more lenient, but different types of experience had quantitatively different magnitudes of impact. In addition, skipping rating sessions, and in that way disconnecting from the rater community, increased the likelihood of a rater to be inconsistent. Finally, we provide methodological recommendations for future research and consider implications for practice.
Keywords: evaluation; languages; language skills; Finnish language
Free keywords: generalized linear mixed models; Many-Facets Rasch model; rater consistency; rater experience; rater severity; speaking assessment
Contributing organizations
Related projects
- ‘Broken Finnish’: Accent perceptions in societal gatekeeping
- Halonen, Mia
- Research Council of Finland
Ministry reporting: Yes
Preliminary JUFO rating: 2
- Accounting (School of Business and Economics JSBE) YLA
- Applied language studies (Centre for Applied Language Studies SOLKI)
- National Certificates of Language Proficiency (Centre for Applied Language Studies SOLKI) YKI
- Multiliteracies for social participation and in learning across the life span (University of Jyväskylä JYU) MultiLEAP; 2021-2026. Formerly RECLAS