ESCA Workshop on Audio-Visual Speech Processing (AVSP'97)

September 26-27, 1997
Rhodes, Greece

Auditory-Visual Interaction in Voice Localization and in Bimodal Speech Recognition: The Effects of Desynchronization

Paul Bertelson (1,2), Jean Vroomenti (2), Beatrice de Gelderti (1,2)

(1) Universite libre de Bruxelles, Bruxelles, Belgium
(2) Tilburg University, The Netherlands

The effects of AV asynchrony on respectively the visual bias of auditory input localization and on the McGurk phenomenon were examined within a single experimental situation. On each trial, the face of a talker, articulating one of the two trisyllables /ama/ or /ana/, or staying still, was shown on a screen and his voice saying one of the two tokens was delivered on a hidden loudspeaker to the left of the right of the screen. The subject pointed to the apparent origin of voice, and repeated the heard utterance. With synchronous presentations or short lags of the auditory input, identification responses were influenced by the nature of the visual input (McGurk effect), and pointing responses were attracted toward the talker's face when it moved, compared with trials on which it did not (visual localization bias). Both effects tended to disappear with larger positive auditory lags or with negative ones. But the relation to lag depended on peculiarities of the presented token for localization, and not for identification.

Full Paper

Bibliographic reference.  Bertelson, Paul / Vroomenti, Jean / Gelderti, Beatrice de (1997): "Auditory-visual interaction in voice localization and in bimodal speech recognition: the effects of desynchronization", In AVSP-1997, 97-100.