Sixth ISCA Workshop on Speech Synthesis

Bonn, Germany
August 22-24, 2007

Analysis Methods for Assessing TTS Intelligibility

H. Timothy Bunnell, Jason Lilley

Center for Pediatric Auditory and Speech Sciences, Nemours Biomedical Research, USA & Department of Linguistics & Cognitive Sciences, University of Delaware, USA

Semantically unpredictable (SU) sentences are often used to assess intelligibility of TTS systems, but analyses of listener responses to SU sentences can be a labor-intensive process. In this paper we compare several approaches to the analysis of data from an SUS task. Data from a study comparing five TTS systems were analyzed in a variety of ways ranging from string edit measures based on carefully hand-corrected phonetically transcribed responses to largely uncorrected words- or sentences-correct measures. Results suggest that a simple sentences-correct measure is adequate when only rank order information is of interest. However, the sentencescorrect measure masks the magnitude of differences between systems and should be avoided when it is important to gage how large the difference in intelligibility is between systems. In preparing response data for analysis, careful human interpretation of listener response data can lead to higher intelligibility measures overall, but does not interact with TTS system or other factors and consequently does not lead to different conclusions when comparing multiple TTS systems. This suggests that largely automated scoring procedures are feasible.

Full Paper

Bibliographic reference.  Bunnell, H. Timothy / Lilley, Jason (2007): "Analysis methods for assessing TTS intelligibility", In SSW6-2007, 374-379.