Understanding and Visualizing Raw Waveform-Based CNNs

Hannah Muckenhirn, Vinayak Abrol, Mathew Magimai-Doss, S├ębastien Marcel

Modeling directly raw waveforms through neural networks for speech processing is gaining more and more attention. Despite its varied success, a question that remains is: what kind of information are such neural networks capturing or learning for different tasks from the speech signal? Such an insight is not only interesting for advancing those techniques but also for understanding better speech signal characteristics. This paper takes a step in that direction, where we develop a gradient based approach to estimate the relevance of each speech sample input on the output score. We show that analysis of the resulting “relevance signal” through conventional speech signal processing techniques can reveal the information modeled by the whole network. We demonstrate the potential of the proposed approach by analyzing raw waveform CNN-based phone recognition and speaker identification systems.

 DOI: 10.21437/Interspeech.2019-2341

Cite as: Muckenhirn, H., Abrol, V., Magimai-Doss, M., Marcel, S. (2019) Understanding and Visualizing Raw Waveform-Based CNNs. Proc. Interspeech 2019, 2345-2349, DOI: 10.21437/Interspeech.2019-2341.

  author={Hannah Muckenhirn and Vinayak Abrol and Mathew Magimai-Doss and S├ębastien Marcel},
  title={{Understanding and Visualizing Raw Waveform-Based CNNs}},
  booktitle={Proc. Interspeech 2019},