ISCA Archive Interspeech 2013
ISCA Archive Interspeech 2013

The albayzin 2012 language recognition evaluation

Luis Javier Rodríguez-Fuentes, Niko Brümmer, Mikel Penagarikano, Amparo Varona, Germán Bordel, Mireia Diez

The Albayzin 2012 Language Recognition Evaluation (LRE), carried out from June to October 2012, was the third effort made by the Spanish/Portuguese community for benchmarking language recognition technology. As in previous Albayzin 2008 and 2010 evaluations, the task consisted on deciding whether or not a target language was spoken in a test utterance. The primary condition involved 6 target languages for which there was plenty of training data: English, Portuguese and the four official languages in Spain (Basque, Catalan, Galician and Spanish). A new challenging condition was defined involving 4 target languages for which no training data were available: French, German, Greek and Italian. In both cases, other (Out-Of-Set) languages were also recorded to allow open-set verification tests. An innovative feature of this evaluation, not common to other evaluations, was that audio data for system development and evaluation were extracted from YouTube videos. Also, a new performance metric was proposed, the so called Multiclass Cross-Entropy, summarizing in a single figure the information provided by system scores, without the need to take hard decisions. This paper presents the main features of the evaluation and analyses the performance of the submitted systems on the different conditions, including the confusion among target languages.

doi: 10.21437/Interspeech.2013-387

Cite as: Rodríguez-Fuentes, L.J., Brümmer, N., Penagarikano, M., Varona, A., Bordel, G., Diez, M. (2013) The albayzin 2012 language recognition evaluation. Proc. Interspeech 2013, 1497-1501, doi: 10.21437/Interspeech.2013-387

  author={Luis Javier Rodríguez-Fuentes and Niko Brümmer and Mikel Penagarikano and Amparo Varona and Germán Bordel and Mireia Diez},
  title={{The albayzin 2012 language recognition evaluation}},
  booktitle={Proc. Interspeech 2013},