Improving On-Device Speaker Verification Using Federated Learning with Privacy

Filip Granqvist, Matt Seigel, Rogier van Dalen, Áine Cahill, Stephen Shum, Matthias Paulik


Information on speaker characteristics can be useful as side information in improving speaker recognition accuracy. However, such information is often private. This paper investigates how privacy-preserving learning can improve a speaker verification system, by enabling the use of privacy-sensitive speaker data to train an auxiliary classification model that predicts vocal characteristics of speakers. In particular, this paper explores the utility achieved by approaches which combine different federated learning and differential privacy mechanisms. These approaches make it possible to train a central model while protecting user privacy, with users’ data remaining on their devices. Furthermore, they make learning on a large population of speakers possible, ensuring good coverage of speaker characteristics when training a model. The auxiliary model described here uses features extracted from phrases which trigger a speaker verification system. From these features, the model predicts speaker characteristic labels considered useful as side information. The knowledge of the auxiliary model is distilled into a speaker verification system using multi-task learning, with the side information labels predicted by this auxiliary model being the additional task. This approach results in a 6% relative improvement in equal error rate over a baseline system.


 DOI: 10.21437/Interspeech.2020-2944

Cite as: Granqvist, F., Seigel, M., Dalen, R.V., Cahill, Á., Shum, S., Paulik, M. (2020) Improving On-Device Speaker Verification Using Federated Learning with Privacy. Proc. Interspeech 2020, 4328-4332, DOI: 10.21437/Interspeech.2020-2944.


@inproceedings{Granqvist2020,
  author={Filip Granqvist and Matt Seigel and Rogier van Dalen and Áine Cahill and Stephen Shum and Matthias Paulik},
  title={{Improving On-Device Speaker Verification Using Federated Learning with Privacy}},
  year=2020,
  booktitle={Proc. Interspeech 2020},
  pages={4328--4332},
  doi={10.21437/Interspeech.2020-2944},
  url={http://dx.doi.org/10.21437/Interspeech.2020-2944}
}