Incremental Machine Speech Chain Towards Enabling Listening While Speaking in Real-Time

Sashi Novitasari, Andros Tjandra, Tomoya Yanagita, Sakriani Sakti, Satoshi Nakamura


Inspired by a human speech chain mechanism, a machine speech chain framework based on deep learning was recently proposed for the semi-supervised development of automatic speech recognition (ASR) and text-to-speech synthesis (TTS) systems. However, the mechanism to listen while speaking can be done only after receiving entire input sequences. Thus, there is a significant delay when encountering long utterances. By contrast, humans can listen to what they speak in real-time, and if there is a delay in hearing, they won’t be able to continue speaking. In this work, we propose an incremental machine speech chain towards enabling machine to listen while speaking in real-time. Specifically, we construct incremental ASR (ISR) and incremental TTS (ITTS) by letting both systems improve together through a short-term loop. Our experimental results reveal that our proposed framework is able to reduce delays due to long utterances while keeping a comparable performance to the non-incremental basic machine speech chain.


 DOI: 10.21437/Interspeech.2020-2034

Cite as: Novitasari, S., Tjandra, A., Yanagita, T., Sakti, S., Nakamura, S. (2020) Incremental Machine Speech Chain Towards Enabling Listening While Speaking in Real-Time. Proc. Interspeech 2020, 4372-4376, DOI: 10.21437/Interspeech.2020-2034.


@inproceedings{Novitasari2020,
  author={Sashi Novitasari and Andros Tjandra and Tomoya Yanagita and Sakriani Sakti and Satoshi Nakamura},
  title={{Incremental Machine Speech Chain Towards Enabling Listening While Speaking in Real-Time}},
  year=2020,
  booktitle={Proc. Interspeech 2020},
  pages={4372--4376},
  doi={10.21437/Interspeech.2020-2034},
  url={http://dx.doi.org/10.21437/Interspeech.2020-2034}
}