FAAVSP - The 1st Joint Conference on Facial Analysis, Animation, and
Auditory-Visual Speech Processing

Vienna, Austria
September 11-13, 2015

Lip Animation Synthesis: a Unified Framework for Speaking and Laughing Virtual Agent

Yu Ding, Catherine Pelachaud

CNRS-LTCI, Télécom-ParisTech, Paris, France

This paper proposes a unified statistical framework to synthesize speaking and laughing lip animations for virtual agents in real time. Our lip animation synthesis model takes as input the decomposition of a spoken text into phonemes as well as their duration. Our model can be used with synthesized speech. First, Gaussian mixture models (GMMs), called lip shape GMMs, are used to model the relationship between phoneme duration and lip shape from human motion capture data; then an interpolation function is learnt from human motion capture data, which is based on hidden Markov models (HMMs), called HMMs interpolation. In the synthesis step, lip shapeGMMs are used to infer a first lip shape stream from the inputs; then this lip shape stream is smoothed by the learnt HMMs interpolation, to obtain the synthesized lip animation. The effectiveness of the proposed framework is confirmed in the objective evaluation. Index Terms: lip animation, speech to animation, interactive virtual agent, laughter, speech, Gaussian mixture models (GMMs), hidden Markov models (HMMs)

Full Paper

Bibliographic reference.  Ding, Yu / Pelachaud, Catherine (2015): "Lip animation synthesis: a unified framework for speaking and laughing virtual agent", In FAAVSP-2015, 78-83.