Second Language Studies: Acquisition, Learning, Education and Technology

Tokyo, Japan
September 22-24, 2010

How Many Labellers? Modelling Inter-Labeller Agreement and System Performance for the Automatic Assessment of Non-Native Prosody

Florian Hönig (1), Anton Batliner (1), Karl Weilhammer (2), Elmar Nöth (1)

(1) Pattern Recognition Lab, Universität Erlangen-Nürnberg, Germany
(2) Digital publishing, München, Germany

On a database of non-native English productions annotated by 60 native English speakers as for their quality w. r. t. intelligibility, non-native accent, melody and rhythm, we study how inter-labeller correlation and performance of a regression system change when varying the number of labellers used for training. This depends highly on the difficulty of the labelling task, the features used by the regression system and the type of regression used. We propose a model that parametrises these dependencies and is able to predict the system’s performance when increasing the number of labellers. This can provide a valuable basis for decision-making when trying to improve an existing regression system as efficiently as possible. We show the plausibility of our approach by experimental evaluation.

Full Paper

Bibliographic reference.  Hönig, Florian / Batliner, Anton / Weilhammer, Karl / Nöth, Elmar (2010): "How many labellers? modelling inter-labeller agreement and system performance for the automatic assessment of non-native prosody", In L2WS-2010, paper O2-5.