5th European Conference on Speech Communication and Technology

Rhodes, Greece
September 22-25, 1997

Generic Template for the Evaluation of Dialogue Management Systems

Gavin E. Churcher, Eric S. Atwell, Clive Souter

Centre for Computer Analysis of Language And Speech (CCALAS), Artificial Intelligence Division, School of Computer Studies The University of Leeds, Leeds, Yorkshire, England

We present a generic template for spoken dialogue systems integrating speech recognition and synthesis with 'higher-level' natural language dialogue modelling components. The generic model is abstracted from a number of real application systems targetted at very different domains. Our research aim in developing this generic template is to investigate a new approach to the evaluation of Dialogue Management Systems. Rather than attempting to measure accuracy/speed of output, we propose principles for the evaluation of the underlying theoretical linguistic model of Dialogue Management in a given system, in terms of how well it fits our generic template for Dialogue Management Systems. This is a measure of 'genericness' or 'application-independence' of a given system, which can be used to moderate accuracy/speed scores in comparisons of very unlike DMSs serving different domains. This relates to (but is orthogonal to) Dialogue Management Systems evaluation in terms of naturalness and like measurable metrics; it follows more closely emerging qualitative evaluation techniques for NL grammatical parsing schemes.

Full Paper

Bibliographic reference.  Churcher, Gavin E. / Atwell, Eric S. / Souter, Clive (1997): "Generic template for the evaluation of dialogue management systems", In EUROSPEECH-1997, 2247-2250.