4th International Conference on Spoken Language Processing

Philadelphia, PA, USA
October 3-6, 1996

Audio-visual Speech Perception Without Speech Cues

Helena M. Saldaņa (1), David B. Pisoni (1), Jennifer M. Fellowes (2), Robert E. Remez (2)

(1) Speech Research Laboratory, Indiana University, Bloomington, IN, USA
(2) Psychology Department, Barnard College, New York, NY, USA

A series of experiments was conducted in which listeners were presented with audio-visual sentences in a transcription task. The visual components of the stimuli consisted of a male talker’s face. The acoustic components consisted of: (1) natural speech (2) envelope-shaped noise which preserved the duration and amplitude of the original speech waveform and (3) various types of sinewave speech signals that followed the formant frequencies of a natural utterance. Sinewave speech is a skeletonized version of a natural utterance which contains frequency and amplitude variation of the formants, but lacks any fine-grained acoustic structure of speech. Intelligibility of the present set of sinewave sentences was relatively low in contrast to previous findings (Remez, Rubin, Pisoni, & Carrell 1981). However, intelligibility was greatly increased when visual information from a talkers face was presented along with the auditory stimuli. Further experiments demonstrated that the intelligibility of single tones increased differentially depending on which formant analog was presented. It was predicted that the increase in intelligibility for the sinewave speech with an added video display would be greater than the gain observed with envelope-shaped noise. This prediction is based on the assumption that the information-bearing phonetic properties of spoken utterances are preserved in the audio+visual sine-wave conditions. This prediction was borne out for the tonal analog of the second formant (T2), but not the tonal analog of the first formant (T1) or third formant (T3), suggesting that the information contained in the T2 analog is relevant for audiovisual integration.

Full Paper

Bibliographic reference.  Saldaņa, Helena M. / Pisoni, David B. / Fellowes, Jennifer M. / Remez, Robert E. (1996): "Audio-visual speech perception without speech cues", In ICSLP-1996, 2187-2190.