A1 Journal article (refereed)
Top-Down Predictions of Familiarity and Congruency in Audio-Visual Speech Perception at Neural Level (2019)

Kolozsvári, O. B., Xu, W., Leppänen, P. H. T., & Hämäläinen, J. A. (2019). Top-Down Predictions of Familiarity and Congruency in Audio-Visual Speech Perception at Neural Level. Frontiers in Human Neuroscience, 13, Article 243. https://doi.org/10.3389/fnhum.2019.00243

JYU authors or editors

Publication details

All authors or editors: Kolozsvári, Orsolya B.; Xu, Weiyong; Leppänen, Paavo H. T.; Hämäläinen, Jarmo A.

Journal or series: Frontiers in Human Neuroscience

eISSN: 1662-5161

Publication year: 2019

Volume: 13

Article number: 243

Publisher: Frontiers Media

Publication country: Switzerland

Publication language: English

DOI: https://doi.org/10.3389/fnhum.2019.00243

Publication open access: Openly available

Publication channel open access: Open Access channel

Publication is parallel published (JYX): https://jyx.jyu.fi/handle/123456789/65130

Abstract

During speech perception, listeners rely on multimodal input and make use of both auditory and visual information. When presented with speech, for example syllables, the differences in brain responses to distinct stimuli are not, however, caused merely by the acoustic or visual features of the stimuli. The congruency of the auditory and visual information and the familiarity of a syllable, that is, whether it appears in the listener's native language or not, also modulates brain responses. We investigated how the congruency and familiarity of the presented stimuli affect brain responses to audio-visual (AV) speech in 12 adult Finnish native speakers and 12 adult Chinese native speakers. They watched videos of a Chinese speaker pronouncing syllables (/pa/, /pha/, /ta/, /tha/, /fa/) during a magnetoencephalography (MEG) measurement where only /pa/ and /ta/ were part of Finnish phonology while all the stimuli were part of Chinese phonology. The stimuli were presented in audio-visual (congruent or incongruent), audio only, or visual only conditions. The brain responses were examined in five time-windows: 75-125, 150-200, 200-300, 300-400, and 400-600 ms. We found significant differences for the congruency comparison in the fourth time-window (300-400 ms) in both sensor and source level analysis. Larger responses were observed for the incongruent stimuli than for the congruent stimuli. For the familiarity comparisons no significant differences were found. The results are in line with earlier studies reporting on the modulation of brain responses for audio-visual congruency around 250-500 ms. This suggests a much stronger process for the general detection of a mismatch between predictions based on lip movements and the auditory signal than for the top-down modulation of brain responses based on phonological information.

Keywords: perception (activity); stimuli (role related to effect); speech (phenomena); MEG

Free keywords: speech perception; magnetoencephalography; audio-visual stimuli; audio-visual integration; familiarity

Contributing organizations

Related projects

Ministry reporting: Yes

Reporting Year: 2019

JUFO rating: 1