Auditory-Visual Speech Processing (AVSP) 2010

Hakone, Kanagawa, Japan
September 30-October 3, 2010

Optimized Photorealistic Audiovisual Speech Synthesis Using Active Appearance Modeling

Wesley Mattheyses, Lukas Latacz, Werner Verhelst

Vrije Universiteit Brussel, Dept. ETRO-DSSP, Interdisciplinary Institute for Broadband Technology IBBT, Brussels, Belgium

Active appearance models can represent image information in terms of shape and texture parameters. This paper explains why this makes them highly suitable for data-based 2D audiovisual text-to-speech synthesis. We elaborate on how the differentiation between shape and texture information can be fully exploited to create appropriate unit-selection costs and to enhance the video concatenations. The latter is very important since for the synthetic visual speech a careful balancing between signal smoothness and articulation strength is required. Several optimization strategies to enhance the quality of the synthetic visual speech are proposed. By measuring the properties of each model parameter, an effective normalization of the visual speech database is feasible. In addition, the visual joins can be optimized by a parameter-specific concatenation smoothing. To further enhance the naturalness of the synthetic speech, a spectrum-based smoothing approach is introduced.

Index Terms: audiovisual speech synthesis, AAM modeling

Full Paper

Bibliographic reference.  Mattheyses, Wesley / Latacz, Lukas / Verhelst, Werner (2010): "Optimized photorealistic audiovisual speech synthesis using active appearance modeling", In AVSP-2010, paper S8-1.