Whisper Augmented End-to-End/Hybrid Speech Recognition System — CycleGAN Approach

Prithvi R.R. Gudepu, Gowtham P. Vadisetti, Abhishek Niranjan, Kinnera Saranu, Raghava Sarma, M. Ali Basha Shaik, Periyasamy Paramasivam


Automatic speech recognition (ASR) systems are known to perform poorly under whispered speech conditions. One of the primary reasons is the lack of large annotated whisper corpora. To address this challenge, we propose data augmentation with synthetic whisper corpus generated from normal speech using Cycle-Consistent Generative Adversarial Network (CycleGAN). We train CycleGAN model with a limited corpus of parallel whispered and normal speech, aligned using Dynamic Time Warping (DTW). The model learns frame-wise mapping from feature vectors of normal speech to those of whisper. We then augment ASR systems with the generated synthetic whisper corpus. In this paper, we validate our proposed approach using state-of-the-art end-to-end (E2E) and hybrid ASR systems trained on publicly available Librispeech, wTIMIT and internally recorded far-field corpora. We achieved 23% relative reduction in word error rate (WER) compared to baseline on whisper test sets. In addition, we also achieved WER reductions on Librispeech and far-field test sets.


 DOI: 10.21437/Interspeech.2020-2639

Cite as: Gudepu, P.R., Vadisetti, G.P., Niranjan, A., Saranu, K., Sarma, R., Shaik, M.A.B., Paramasivam, P. (2020) Whisper Augmented End-to-End/Hybrid Speech Recognition System — CycleGAN Approach. Proc. Interspeech 2020, 2302-2306, DOI: 10.21437/Interspeech.2020-2639.


@inproceedings{Gudepu2020,
  author={Prithvi R.R. Gudepu and Gowtham P. Vadisetti and Abhishek Niranjan and Kinnera Saranu and Raghava Sarma and M. Ali Basha Shaik and Periyasamy Paramasivam},
  title={{Whisper Augmented End-to-End/Hybrid Speech Recognition System — CycleGAN Approach}},
  year=2020,
  booktitle={Proc. Interspeech 2020},
  pages={2302--2306},
  doi={10.21437/Interspeech.2020-2639},
  url={http://dx.doi.org/10.21437/Interspeech.2020-2639}
}