A4 Article in conference proceedings
Developing Automated Feedback on Spoken Performance : Exploring the Functioning of Five Analytic Rating Scales Using Many-facet Rasch Measurement  (2022)

Zansen, A. V., & Huhta, A. (2022). Developing Automated Feedback on Spoken Performance : Exploring the Functioning of Five Analytic Rating Scales Using Many-facet Rasch Measurement . In J. H. Jantunen, J. Kalja-Voima, M. Laukkarinen, A. Puupponen, M. Salonen, T. Saresma, J. Tarvainen, & S. Ylönen (Eds.), Diversity of Methods and Materials in Digital Human Sciences : Proceedings of the Digital Research Data and Human Sciences DRDHum Conference 2022, December 1-3, Jyväskylä, Finland (pp. 211-229). Jyväskylän yliopisto. http://urn.fi/URN:ISBN:978-951-39-9450-1

JYU authors or editors

Publication details

All authors or editors: Zansen, Anna von; Huhta, Ari

Parent publication: Diversity of Methods and Materials in Digital Human Sciences : Proceedings of the Digital Research Data and Human Sciences DRDHum Conference 2022, December 1-3, Jyväskylä, Finland

Parent publication editors: Jantunen, Jarmo Harri; Kalja-Voima, Johanna; Laukkarinen, Matti; Puupponen, Anna; Salonen, Margareta; Saresma, Tuija; Tarvainen, Jenny; Ylönen, Sabine

Conference:

Place and date of conference: Jyväskylä, Suomi, 1.-3.12.2022

eISBN: 978-951-39-9450-1

Publication year: 2022

Pages range: 211-229

Number of pages in the book: 243

Publisher: Jyväskylän yliopisto

Publication country: Finland

Publication language: English

Persistent website address: http://urn.fi/URN:ISBN:978-951-39-9450-1

Publication open access: Openly available

Publication channel open access: Open Access channel

Publication is parallel published (JYX): https://jyx.jyu.fi/handle/123456789/84333

Abstract

In this study, we used the Many-facet Rasch measurement (MFRM) to explore the quality of ratings as well as the functioning of five analytic rating scales developed for automated assessment of L2 speech. This study is part of a multidisciplinary research project that develops automatic speech recognition (ASR), automated scoring and automated feedback for L2 Finnish and Swedish. The data include the analytic ratings (task completion, fluency, pronunciation, range, accuracy) gathered from human raters (n=14) who assessed L2 Finnish learners’ (n=64) speech samples using Moodle. The four-facet Rasch analysis showed that the raters performed and the rating scales functioned well, although task completion seems to be more challenging to apply consistently than the other criteria. Moreover, it proved to be more difficult to receive a certain score on some dimensions, namely fluency and range, than others. The study has implications for score reporting. We demonstrated that a) the different analytical rating scales have somewhat different structure, b) scores do not advance with equal intervals and c) a certain score on a certain dimension might require a bigger leap forward in ability than on other dimensions. The results will be used for designing encouraging and accurate automated feedback to L2 Finnish and Swedish learners.

Keywords: second language; Finnish language; Swedish language; oral language skills; phonetics; evaluation; feedback; evaluation methods; speech recognition; educational technology; Rasch model

Free keywords: automated feedback; language assessment; rating scales; oral skills

Contributing organizations

Related projects

Ministry reporting: Yes

VIRTA submission year: 2022

JUFO rating: 0