Evaluating the Reliability of Acoustic Speech Embeddings

Robin Algayres, Mohamed Salah Zaiem, Benoît Sagot, Emmanuel Dupoux


Speech embeddings are fixed-size acoustic representations of variable-length speech sequences. They are increasingly used for a variety of tasks ranging from information retrieval to unsupervised term discovery and speech segmentation. However, there is currently no clear methodology to compare or optimize the quality of these embeddings in a task-neutral way. Here, we systematically compare two popular metrics, ABX discrimination and Mean Average Precision (MAP), on 5 languages across 17 embedding methods, ranging from supervised to fully unsupervised, and using different loss functions (autoencoders, correspondance autoencoders, siamese). Then we use the ABX and MAP to predict performances on a new downstream task: the unsupervised estimation of the frequencies of speech segments in a given corpus. We find that overall, ABX and MAP correlate with one another and with frequency estimation. However, substantial discrepancies appear in the fine-grained distinctions across languages and/or embedding methods. This makes it unrealistic at present to propose a task-independent silver bullet method for computing the intrinsic quality of speech embeddings. There is a need for more detailed analysis of the metrics currently used to evaluate such embeddings.


 DOI: 10.21437/Interspeech.2020-2362

Cite as: Algayres, R., Zaiem, M.S., Sagot, B., Dupoux, E. (2020) Evaluating the Reliability of Acoustic Speech Embeddings. Proc. Interspeech 2020, 4621-4625, DOI: 10.21437/Interspeech.2020-2362.


@inproceedings{Algayres2020,
  author={Robin Algayres and Mohamed Salah Zaiem and Benoît Sagot and Emmanuel Dupoux},
  title={{Evaluating the Reliability of Acoustic Speech Embeddings}},
  year=2020,
  booktitle={Proc. Interspeech 2020},
  pages={4621--4625},
  doi={10.21437/Interspeech.2020-2362},
  url={http://dx.doi.org/10.21437/Interspeech.2020-2362}
}