The Seventh ISCA Tutorial and Research Workshop on Speech Synthesis

Kyoto, Japan
September 22-24, 2010

Further Exploration of the Possibilities and Pitfalls of Multidimensional Scaling as a Tool for the Evaluation of the Quality of Synthesized Speech

Anna C. Janska (1), Robert A. J. Clark (2)

(1) IMPRS NeuroCom, University of Leipzig, Germany
(2) CSTR, The University of Edinburgh, U.K.

Multidimensional scaling (MDS) has been suggested as a useful tool for the evaluation of the quality of synthesized speech. However, it has not yet been extensively tested for its application in this specific area of evaluation. In a series of experiments based on data from the Blizzard Challenge 2008 the relations betweenWeighted Euclidean Distance Scaling and Simple Euclidean Distance Scaling is investigated to understand how aggregating data affects the MDS configuration. These results are compared to those collected as mean opinion scores (MOS). The ranks correspond, and MOS can be predicted from an object’s space in the MDS generated stimulus space. The big advantage of MDS over MOS is its diagnostic value; dimensions along which stimuli vary are not correlated, as is the case in modular evaluation using MOS. Finally, it will be attempted to generalize from the MDS representations of the thoroughly tested subset to the aggregated data of the larger-scale Blizzard Challenge.

Full Paper

Bibliographic reference.  Janska, Anna C. / Clark, Robert A. J. (2010): "Further exploration of the possibilities and pitfalls of multidimensional scaling as a tool for the evaluation of the quality of synthesized speech", In SSW7-2010, 142-147.