Rapid RNN-T Adaptation Using Personalized Speech Synthesis and Neural Language Generator

Yan Huang, Jinyu Li, Lei He, Wenning Wei, William Gale, Yifan Gong


Rapid unsupervised speaker adaptation in an E2E system posits us new challenges due to its end-to-end unified structure in addition to its intrinsic difficulty of data sparsity and imperfect label [1]. Previously we proposed utilizing the content relevant personalized speech synthesis for rapid speaker adaptation and achieved significant performance breakthrough in a hybrid system [2]. In this paper, we answer the following two questions: First, how to effectively perform rapid speaker adaptation in an RNN-T. Second, whether our previously proposed approach is still beneficial for the RNN-T and what are the modification and distinct observations. We apply the proposed methodology to a speaker adaptation task in a state-of-art presentation transcription RNN-T system. In the 1 min setup, it yields 11.58% or 7.95% relative word error rate (WER) reduction for the sup/unsup adaptation, comparing to the negligible gain when adapting with 1 min source speech. In the 10 min setup, it yields 15.71% or 8.00% relative WER reduction, doubling the gain of the source speech adaptation. We further apply various data filtering techniques and significantly bridge the gap between sup/unsup adaptation.


 DOI: 10.21437/Interspeech.2020-1290

Cite as: Huang, Y., Li, J., He, L., Wei, W., Gale, W., Gong, Y. (2020) Rapid RNN-T Adaptation Using Personalized Speech Synthesis and Neural Language Generator. Proc. Interspeech 2020, 1256-1260, DOI: 10.21437/Interspeech.2020-1290.


@inproceedings{Huang2020,
  author={Yan Huang and Jinyu Li and Lei He and Wenning Wei and William Gale and Yifan Gong},
  title={{Rapid RNN-T Adaptation Using Personalized Speech Synthesis and Neural Language Generator}},
  year=2020,
  booktitle={Proc. Interspeech 2020},
  pages={1256--1260},
  doi={10.21437/Interspeech.2020-1290},
  url={http://dx.doi.org/10.21437/Interspeech.2020-1290}
}