5th International Conference on Spoken Language Processing

Sydney, Australia
November 30 - December 4, 1998

Speech Driven 3-D Face Point Trajectory Synthesis Algorithm

Levent M. Arslan, David Talkin

Entropic Inc., USA

A novel algorithm is proposed which generates three-dimensional face point trajectories for a given speech file. The proposed algorithm first employs an off-line training phase. In this phase, recorded face point trajectories along with their speech data and phonetic labels are used to generate phonetic codebooks. These codebooks consist of both acoustic and visual features. During the synthesis stage, speech input is rated in terms of its similarity to the codebook entries, and a weight is assigned to each codebook entry. If the phonetic information about the test speech is available, this is utilized in restricting the codebook search to only several codebook entries which are visually closest to the current phoneme. These weights are then used to synthesize the principal components of the face point trajectory. The performance of the algorithm is tested on held-out data, and the synthesized face point trajectories showed a correlation of 0.73 with true face point trajectories.

Full Paper

Bibliographic reference.  Arslan, Levent M. / Talkin, David (1998): "Speech driven 3-d face point trajectory synthesis algorithm", In ICSLP-1998, paper 0110.