Fourth European Conference on Speech Communication and Technology

Madrid, Spain
September 18-21, 1995

A Method To Quantify the Error Distribution in Confusion Matrices

Rob J. J. H. van Son

Institute of Phonetic Sciences, University of Amsterdam, Amsterdam, The Netherlands

Currently the error-rate is the most widely used general measure of performance for identification experiments. However, the error-rate gives no information about the distribution of errors over the available response categories. Starting from the definition of perplexity of the observations in a confusion matrix, a new measure is introduced, the error-dispersion, that is normalized with respect to error-rate. The error-dispersion can be interpreted as the effective number of error categories per stimulus or response. Furthermore, a technique is introduced to estimate the difference between confusion matrices as a fraction of unique observations or errors. With examples from the literature, it is shown that error-dispersion can point out similarities in cases where error-rate varies, and can point out differences when error-rates are similar.

Full Paper

Bibliographic reference.  Son, Rob J. J. H. van (1995): "A method to quantify the error distribution in confusion matrices", In EUROSPEECH-1995, 2277-2280.