Visually Grounded Learning of Keyword Prediction from Untranscribed Speech

Herman Kamper, Shane Settle, Gregory Shakhnarovich, Karen Livescu


During language acquisition, infants have the benefit of visual cues to ground spoken language. Robots similarly have access to audio and visual sensors. Recent work has shown that images and spoken captions can be mapped into a meaningful common space, allowing images to be retrieved using speech and vice versa. In this setting of images paired with untranscribed spoken captions, we consider whether computer vision systems can be used to obtain textual labels for the speech. Concretely, we use an image-to-words multi-label visual classifier to tag images with soft textual labels, and then train a neural network to map from the speech to these soft targets. We show that the resulting speech system is able to predict which words occur in an utterance — acting as a spoken bag-of-words classifier — without seeing any parallel speech and text. We find that the model often confuses semantically related words, e.g. “man” and “person”, making it even more effective as a semantic keyword spotter.


 DOI: 10.21437/Interspeech.2017-502

Cite as: Kamper, H., Settle, S., Shakhnarovich, G., Livescu, K. (2017) Visually Grounded Learning of Keyword Prediction from Untranscribed Speech. Proc. Interspeech 2017, 3677-3681, DOI: 10.21437/Interspeech.2017-502.


@inproceedings{Kamper2017,
  author={Herman Kamper and Shane Settle and Gregory Shakhnarovich and Karen Livescu},
  title={Visually Grounded Learning of Keyword Prediction from Untranscribed Speech},
  year=2017,
  booktitle={Proc. Interspeech 2017},
  pages={3677--3681},
  doi={10.21437/Interspeech.2017-502},
  url={http://dx.doi.org/10.21437/Interspeech.2017-502}
}