5th International Conference on Spoken Language Processing

Sydney, Australia
November 30 - December 4, 1998

Total Quality Evaluation of Speech Synthesis Systems

Jialu Zhang (1), Shiwei Dong (2), Ge Yu (1)

(1) Institute of Acoustics, Academia Sinica, China
(2) Institute of Acoustics,Academia Sinica, China

Based on the performance assessment of speech synthesis systems for Chinese the total quality evaluation of them has been carried out regular since 1994. The total quality evaluation includes speech intelligibility test at different levels (syllable, word and sentence), speech naturalness test and anti-interference ability test for phonetic module and text processing ability test for linguistic module. The designing principle of testing materials and testing methods are briefly described and the test results of four text-to-speech (TTS) systems for Chinese are presented in this paper. It is shown that 1. All professional technicians joining the speech naturalness test with the testing crew together overestimated the overall quality of the four tested systems; 2. The word intelligibility and Semantically Unpredicted Sentence (SUS) score are good for evaluating speech synthesis systems; 3. The anti-interference ability of synthetic speech is rather weak about 20 per cent in syllable intelligibility lower than natural speech under condition of S/N=5dB.

Full Paper

Bibliographic reference.  Zhang, Jialu / Dong, Shiwei / Yu, Ge (1998): "Total quality evaluation of speech synthesis systems", In ICSLP-1998, paper 0060.