13th Annual Conference of the International Speech Communication Association

Portland, OR, USA
September 9-13, 2012

Assessment of User Simulators for Spoken Dialogue Systems by Means of Subspace Multidimensional Clustering

Zoraida Callejas (1), David Griol (2), Klaus-Peter Engelbrecht (3)

(1) Department of Languages and Computer Systems, University of Granada, Granada, Spain
(2) Department of Computer Science, University Carlos III of Madrid, Leganés, Spain
(3) Quality and Usability Lab, Deutsche Telekom Laboratories, TU Berlin, Berlin, Germany

The assessment of user simulators in terms of their similarity with real users implies processing and interpreting large dialogue corpora, for which many interaction parameters can be considered. In this setting, the high dimensionality of the data makes it difficult to compare the dialogues as it is not always appropriate to consider all features equally in order to carry out meaningful interpretations. We propose to use subspace clustering for the assessment of users simulators, as this technique has been successfully applied to tackle and classify high-dimensional information in other areas of study. We created and assessed a user simulator for the Let's Go spoken dialogue system. The experimental results show that the proposed approach is easy to set up and helps to better interpret whether the user simulator has similar behaviours to real human users by creating clusters with different dimensions which cannot be identified with plain clustering techniques.

Index Terms: user simulation, spoken dialogue systems, evaluation, clustering.

Full Paper

Bibliographic reference.  Callejas, Zoraida / Griol, David / Engelbrecht, Klaus-Peter (2012): "Assessment of user simulators for spoken dialogue systems by means of subspace multidimensional clustering", In INTERSPEECH-2012, 250-253.