13th Annual Conference of the International Speech Communication Association

Portland, OR, USA
September 9-13, 2012

Audiovisual Discrimination of CV Syllables: A Simultaneous fMRI-EEG Study

Cyril Dubois (1,2), Rudolph Sock (2)

(1) Romanisches Seminar, University of Zürich, Zürich, Switzerland
(2) Institut de Phonétique de Strasbourg - Equipe de Recherche Parole et Cognition, University of Strasbourg, France

We carried out a simultaneous fMRI-EEG experiment based on discriminating syllabic minimal pairs involving three phonological contrasts characterized by different degrees of visual distinctiveness (vocalic labialization, consonantal place of articulation or voicing). Audiovisual CV syllable pairs were presented either with a static facial configuration or with a dynamic display of articulatory movements. In the sounddisturbed MRI environment, the significant improvement of syllabic discrimination achieved in the dynamic audiovisual modality, compared to the static audiovisual modality was associated with activation of the occipito-temporal cortex (MT + V5) bilaterally, and of the left premotor cortex. MT + V5 was activated in response to facial movements independently of their relation to speech, the latter was specifically activated by phonological discrimination. Significant ERP's to syllabic discrimination were recorded around 150 and 250 ms. Our results provide arguments for the involvement of the speech motor cortex in phonological discrimination, and suggest a multimodal representation of speech units.

Index Terms: Visual distinctiveness, motor cortex, MT+ V5, multimodal representation

Full Paper

Bibliographic reference.  Dubois, Cyril / Sock, Rudolph (2012): "Audiovisual discrimination of CV syllables: a simultaneous fMRI-EEG study", In INTERSPEECH-2012, 410-413.